Systems and methods for representing acoustic signatures from a target scene

ABSTRACT

Acoustic imaging systems can include an acoustic sensing array, an electromagnetic imaging tool, a display, and an audio device. A processor can receive data from the acoustic sensor array and the electromagnetic imaging tool to generate a display image combining acoustic image data and electromagnetic image data. Systems can include an audio device that receives an audio output from the processor and outputs audio feedback signals to a user. The audio feedback signals can represent acoustic signals from an acoustic scene. Systems can provide a display image to a user including acoustic image data, and a user can select an acoustic signal for which to provide a corresponding audio output to an audio device. Audio outputs and display images can change dynamically in response to a change in pointing of the acoustic sensing array, such as by changing a stereo audio output.

RELATED MATTERS

This application is a National Stage filing from International PatentApplication No. PCT/US2019/043230, filed Jul. 24, 2019, which claims thebenefit of U.S. Provisional Application No. 62/702,716 filed Jul. 24,2018, the contents of which are hereby incorporated by reference.

BACKGROUND

Presently available acoustic imaging devices include acoustic sensorarray configurations that have various frequency sensitivity limitationsdue to a variety of factors. For instance, some acoustic imaging devicesare designed to be responsive to a range of acoustic frequencies betweenapproximately 20 Hz and approximately 20 kHz. Other devices (e.g.,ultrasonic devices) are designed to be responsive to a range of acousticfrequencies between approximately 38 kHz and approximately 45 kHz.

However, acoustic imaging devices that are generally designed operatingin the 20 Hz to 20 kHz frequency range cannot effectively detect orimage higher frequencies, for example, up to or above approximately 50kHz. Likewise, acoustic or ultrasonic devices that are designed tooperate in the 20 kHz to 50 kHz frequency range cannot effectivelydetect and/or image lower frequencies, for example, at or below 20 kHz.This can be for a variety of reasons. For example, sensor arrays whichare optimized for lower (e.g., audible) frequencies typically containindividual sensors that are farther apart than do sensor arrays that areoptimized for higher (e.g., ultrasonic) frequencies.

Additionally or alternatively to hardware concerns, differentcalculation algorithms and methods of acoustic imaging are often bettersuited for acoustic signals having different frequencies and/ordifferent distances to target, making it difficult to determine how tobest to acoustically image a scene without, particularly to aninexperienced user.

Such discrepancies in imaging different acoustic frequency ranges aredue, in part, to the physics behind the propagation of sound waves ofdifferent frequencies and wavelengths through air. Certain arrayorientations, array sizes, and calculation methods can generally bebetter suited for acoustic signals having different frequencycharacteristics (e.g., audible frequencies, ultrasonic frequencies,etc.).

Similarly, different array properties and/or calculation methods can bebetter suited for acoustic scenes at different distances to target. Forexample, near field acoustic holography for targets at very closedistances, various acoustic beamforming methods for targets at greaterdistances.

Accordingly, acoustic inspection using acoustic arrays (e.g., foracoustic imaging) can require a wide range of equipment, for example,for analysis of acoustic signals having different frequency ranges aswell as expertise in understanding when different hardware andcalculation techniques are appropriate for performing acoustic analysis.This can make acoustic inspections time- and cost-intensive, and canrequire an expert to perform such inspections.

For example, a user may be forced to manually select various hardwareand/or software for performing acoustic analysis. However, aninexperienced analyst may be incapable of knowing the preferredcombination of hardware and software for a given acoustic analysisand/or acoustic scene. Additionally, isolating a sound of interest fromwithin a scene can provide its own challenges, particularly in acluttered scene, and may prove tedious and frustrating to aninexperienced user. For instance, a given acoustic scene, especially ina noisy environment, can include acoustic signals including any numberof frequency, intensity, or other characteristics that may obscureacoustic signals of interest.

Traditional systems often require users to manually identify variousacoustic parameters of interest prior to inspection in order to analyzethe sounds of interest. However, an inexperienced user may be unaware ofhow to best isolate and/or identify various sounds of interest.

Additionally, when multiple imaging technologies (e.g., visible light,infrared, ultraviolet, acoustic, or other imaging techniques) are usedin tandem while inspecting the same object or scene, the physicalplacement and or other settings (e.g., focus position) of the tools usedto perform the different imaging techniques can impact the analysis. Forexample, different locations and/or focus positions of each imagingdevice can result in a parallax error wherein the resulting images maybe misaligned. This may result in inability to properly localize areasof interest and/or problem areas within a scene, documentation errors,and misdiagnosis of problems. For example, with respect to acousticimage data, it can be difficult to identify a location or source of anacoustic signal of interest if acoustic image data is misaligned withrespect to image data from other imaging technologies (e.g., visiblelight and/or infrared image data).

Existing ultrasonic test and inspection tools employ ultrasonicsensor(s), with or without the use of a parabolic dish in order toassist in focusing the sound towards the receiving sensor(s).

When a sound of a specific frequency is detected, it is typicallydisplayed as a rising or falling numerical value, or on a frequency ordecibel level graph on the display of the device. This can be veryconfusing and non-intuitive to many users.

However, isolating, localizing, and analyzing a specific sound can be atedious process, and can be confusing for many end users. The complexand non-intuitive interface between device and human can become abarrier to effective use of the device, and/or require the need foradditional training even to operate basic functionality on the device

Additionally, some of these devices utilize an optional set of audioheadphones which either amplify and broadcast the actual sound, or aheterodyne related to the actual sound being detected. The sound willtypically increase in intensity to the listener's ear as the sensor(s)is aimed closer to the source or the sound, or the sensor(s) are broughtcloser to the sound.

However, isolating a specific sound can be a tedious process, and can beconfusing for many end users. It is often difficult for the user toproperly coordinate the audible or heterodyne increase in the soundintensity with the rapid change in values on the display screen.

SUMMARY

Some aspects of the disclosure relate to acoustic analysis systemscomprising an acoustic sensor array having a plurality of acousticsensor elements and an electromagnetic imaging tool configured toreceive electromagnetic radiation from a target scene and outputelectromagnetic image data representative of the receivedelectromagnetic radiation. Systems can include a display and an audiodevice configured to output audio feedback signals. In some examples,audio the device can include a speaker or headphones, and can be inwired or wireless communication with other system components.

Systems can include a processor in communication with the acousticsensor array, the electromagnetic imaging tool, the display, and theaudio device. In some examples, the processor is configured to receiveelectromagnetic image data from the electromagnetic image tool andacoustic data from the acoustic sensor array. The processor can beconfigured to generate acoustic image data based on the receivedacoustic data, and generate a display image comprising combined acousticimage data and electromagnetic image data.

In some embodiments, a processor is further configured to generate anaudio output representative of the received acoustic data and output theaudio output to the audio device. In an example, the processor isconfigured to detect an acoustic signal and the generated audio outputincludes a representation of the detected acoustic signal. For example,a processor can determine one or more acoustic parameters associatedwith the detected acoustic signal and generate an audio output havingone or more acoustic parameters in common with the detected acousticsignal.

In some examples, the processor is configured to generate a stereo audiooutput having a first audio output to a first audio device and a secondaudio output to a second audio device. In some such examples, the firstaudio device outputs a first audio feedback signal and the second audiodevice outputs a second audio feedback signal. The processor can beconfigured to determine a location of an acoustic signal in the acousticscene and generate the first audio output and the second audio outputbased on the determined location of the acoustic signal in the acousticscene.

In some examples, the processor can detect a plurality of acousticsignals within the scene. The processor can output an audio outputrepresenting a combination fo each of the acoustic signals. Theprocessor can output a plurality of audio outputs, each representativeof one of the plurality of acoustic signals, for example, in a sequence,cycling through a corresponding plurality of audio outputs. In someexamples, a user may select an acoustic signal observed in a displayimage, such as via a touchscreen input. The processor can output anaudio output corresponding to the selected acoustic signal.

In some examples, a processor can be configured to provide a dynamicallychanging output representing an acoustic signal.

For example, an audio output can increase in intensity as an acousticsignal moves toward the center of an acoustic scene. Similarly, in astereo embodiment having first and second audio outputs, an intensity ofthe audio feedback signal representing the acoustic signal in the firstaudio output and an intensity of the audio feedback signal representingthe acoustic signal in the second audio output changes as the pointingof the acoustic sensor array is adjusted. For example, an acousticsignal on the left-hand side of a scene may be represented more stronglyin a left-hand audio feedback signal than a right-hand audio feedbacksignal.

Additionally or alternatively, acoustic image data may change inintensity depending on the pointing of the acoustic sensor array. Forexample, in some embodiments, an indicator of an acoustic signal inacoustic image data may have a variable transparency in the displayimage, wherein the indicator becomes more transparent the further awaythe location of the acoustic signal is from the center of the acousticscene.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B show front and back views of an example acoustic imagingdevice.

FIG. 2 is a functional block diagram illustrating components of anexample of acoustic analysis system.

FIGS. 3A, 3B, and 3C show schematic diagrams of exemplary acousticsensor array configurations within an acoustic analysis system.

FIGS. 4A and 4B show schematic illustrations of parallax error in thegeneration of a frame of visible light image data and acoustic imagedata.

FIGS. 5A and 5B show parallax correction between a visible light imageand an acoustic image.

FIGS. 5C and 5D are colorized versions of FIGS. 5A and 5B.

FIG. 6 is a process flow diagram showing an exemplary method forgenerating a final image combining acoustic image data andelectromagnetic image data.

FIG. 7 is a process-flow diagram showing an exemplary process forgenerating acoustic image data from received acoustic signals.

FIG. 8 shows an exemplary lookup table for determining an appropriatealgorithm and sensor array for use during an acoustic imaging process.

FIG. 9A is an exemplary plot of frequency content of received image dataover time in an acoustic scene.

FIG. 9B shows an exemplary scene including a plurality of locationsemitting acoustic signals.

FIG. 9C shows a plurality of combined acoustic and visible light imagedata at a plurality of predefined frequency ranges.

FIGS. 10A and 10B are exemplary display images including combinedvisible light image data and acoustic image data.

FIGS. 11A and 11B show exemplary plots of frequency vs. time of acousticdata in an acoustic scene.

FIGS. 12A, 12B, and 12C show multiple exemplary ways for comparingacoustic image data to historical acoustic image data stored in adatabase.

FIG. 13 is a process-flow diagram showing exemplary operation ofcomparing received acoustic image data to a database for objectdiagnostics.

FIG. 14 shows a visualization of acoustic data using a gradientpalettization scheme.

FIG. 15 shows a visualization of acoustic data using a plurality ofshaded concentric circles.

FIG. 16 shows an exemplary visualization including both non-numericinformation and alphanumeric information.

FIG. 17 shows another example visualization including both non-numericinformation and alphanumeric information.

FIG. 18 shows another example visualization including both non-numericinformation and alphanumeric information.

FIG. 19 shows an exemplary visualization showing indicators of differentsize and color representative of different acoustic parameter values.

FIG. 20 shows an exemplary visualization showing a plurality ofindicators having different colors indicative of a severity indicated byacoustic signals from the corresponding locations.

FIG. 21 shows a scene including indicators at a plurality of locationswithin the scene showing acoustic signals meeting a predeterminedcondition in a distinguishing way.

FIG. 22 shows a display image including a plurality of icons positionedwithin the display image indicating recognized acoustic profiles withinthe scene.

FIG. 23 shows another exemplary display image showing acoustic data viaa plurality of indicators using concentric circles and alphanumericinformation representing acoustic intensity associated with each of theacoustic signals.

FIG. 24 shows an example display image having an indicator andadditional alphanumeric information associated with the representedacoustic signal.

FIG. 25A shows a system including a display in which an indicator withina display image is selected and a laser pointer emitting a laser towarda scene.

FIG. 25B shows the display shown in the system view of FIG. 25 .

FIG. 26 shows a display image including an indicator having a gradientpalettization scheme representing acoustic image data and includingacoustic image blending control.

FIG. 27 shows a display image including an indicator having a concentriccircle palettization scheme representing acoustic image data andincluding acoustic image blending control.

FIG. 28 shows a display image including an indicator having a gradientpalettization indicating a location in the scene meeting one or morefilter conditions.

FIG. 29 shows a display image including an indicator having a concentriccircle palettization indicating a location in the scene meeting one ormore filter conditions.

FIG. 30 shows a display image including two indicators, each having agradient palettization indicating a location in the scene meeting adifferent filter condition.

FIG. 31 shows a display interface including a display image and avirtual keyboard.

FIG. 32 shows a display embedded into eyewear that can be worn by a userand display a display image.

FIGS. 33A and 33B show a dynamic display image including indicatorshaving dynamic intensity based on the pointing of an acoustic sensorarray.

FIGS. 34A, 34B, 34C, and 34D show a variety of types of audio devices incommunication with an acoustic imaging device in an acoustic imagingsystem.

FIG. 35 shows an exemplary embodiment wherein an audio device providesaudio feedback to represent the displayed acoustic image data.

FIG. 36 shows an exemplary embodiment wherein an audio device providesselective audio feedback corresponding to a selected source of acousticsignals.

FIG. 37 shows another exemplary embodiment wherein an audio deviceprovides audio feedback to represent the displayed acoustic image dataincluding a plurality of acoustic signals.

FIG. 38 shows an exemplary embodiment in which an audio device providesstereo audio feedback to represent the displayed acoustic image data.

DETAILED DESCRIPTION

FIGS. 1A and 1B show front and back views of an example acoustic imagingdevice. FIG. 1A shows a front side of an acoustic imaging device 100having a housing 102 supporting an acoustic sensor array 104 and anelectromagnetic imaging tool 106. In some embodiments, the acousticsensor array 104 includes a plurality of acoustic sensor elements, eachof the plurality of acoustic sensor elements being configured to receiveacoustic signals from an acoustic scene and output acoustic data basedon the received acoustic signals. The electromagnetic imaging tool 106can be configured to receive electromagnetic radiation from a targetscene and output electromagnetic image data representative of thereceived electromagnetic radiation. The electromagnetic imaging tool 106can be configured to detect electromagnetic radiation in one or more ofa plurality of ranges of wavelengths, such as visible light, infrared,ultraviolet, or the like.

In the illustrated example, the acoustic imaging device 100 includes anambient light sensor 108 and a location sensor 116, such as a GPS. Thedevice 100 includes a laser pointer 110, which in some embodiments,includes a laser distance meter. The device 100 includes a torch 112,which can be configured to emit visible light radiation toward a scene,and an infrared illuminator 118, which can be configured to emitinfrared radiation toward a scene. In some examples, device 100 caninclude an illuminator for illuminating a scene over any range ofwavelengths. Device 100 further includes a projector 114, such as animage reprojector, which can be configured to project a generated imageonto a scene, such as a colorized image, and/or a dot projectorconfigured to project a series of dots onto a scene, for example, todetermine a depth profile of the scene.

FIG. 1B shows a back side of the acoustic imaging device 100. As shown,the device includes display 120, which can present image or other data.In some examples, display 120 comprises a touch screen display. Theacoustic imaging device 100 includes a speaker, which can provide audiofeedback signals to a user, and a wireless interface 124, which canenable wireless communication between the acoustic imaging device 100and an external device. The device further includes controls 126, whichcan include one or more buttons, knobs, dials, switches, or otherinterfacing components to enable a user to interface with the acousticimaging device 100. In some examples, controls 126 and a touchscreendisplay combine to provide a user interface of the acoustic imagingdevice 100.

In various embodiments, acoustic imaging devices need not include everyelement shown in the embodiment of FIGS. 1A and 1B. One or moreillustrated components can be excluded from an acoustic imaging device.In some examples, one or more components shown in the embodiment ofFIGS. 1A and 1B can be included as a part of an acoustic imaging systembut included separately from the housing 102. Such components cancommunicate with other components of an acoustic imaging system viawired or wireless communication techniques, for example, using wirelessinterface 124.

FIG. 2 is a functional block diagram illustrating components of anexample of acoustic analysis system 200. The exemplary acoustic analysissystem 200 of FIG. 2 can include a plurality of acoustic sensors such asmicrophones, MEMS, transducers, etc. arranged in an acoustic sensorarray 202. Such arrays can be one-dimensional, two-dimensional, orthree-dimensional. In various examples, the acoustic sensor array candefine any suitable size and shape. In some examples, acoustic sensorarray 202 includes a plurality of acoustic sensors arranged in a gridpattern such as, e.g., an array of sensor elements arranged in verticalcolumns and horizontal rows. In various examples, acoustic sensor array202 can include an array of vertical columns by horizontal rows of,e.g., 8×8, 16×16, 32×32, 64×64, 128×128, 256×256, etc. Other examplesare possible, and various sensor arrays need not necessarily include thesame number of rows as columns. In some embodiments, such sensors can bepositioned on a substrate, for example, such as a printed circuit board(PCB) substrate.

In the configuration shown in FIG. 2 , a processor 212 in communicationwith the acoustic sensor array 202 can receive acoustic data from eachof the plurality of acoustic sensors. During exemplary operation ofacoustic analysis system 200, processor 212 can communicate withacoustic sensor array 202 to generate acoustic image data. For example,processor 212 can be configured to analyze data received from each of aplurality of acoustic sensors arranged in the acoustic sensor array anddetermine an acoustic scene by “back propagating” acoustic signals tothe source of the acoustic signals. In some embodiments, processor 212can generate a digital “frame” of acoustic image data by identifyingvarious source locations and intensities of acoustic signals across atwo-dimensional scene. By generating a frame of acoustic image data,processor 212 captures an acoustic image of a target scene atsubstantially a given point in time. In some examples, a frame comprisesa plurality of pixels making up the acoustic image, wherein each pixelrepresents a portion of the source scene to which acoustic signals havebeen back-propagated.

Components described as processors within the acoustic analysis system200, including processor 212, may be implemented as one or moreprocessors, such as one or more microprocessors, digital signalprocessors (DSPs), application specific integrated circuits (ASICs),field programmable gate arrays (FPGAs), programmable logic circuitry, orthe like, either alone or in any suitable combination. Processor 212 mayalso include memory that stores program instructions and related datathat, when executed by processor 212, cause acoustic analysis system 200and processor 212 to perform the functions attributed to them in thisdisclosure. Memory may include any fixed or removable magnetic, optical,or electrical media, such as RAM, ROM, CD-ROM, hard or floppy magneticdisks, EEPROM, or the like. Memory may also include a removable memoryportion that may be used to provide memory updates or increases inmemory capacities. A removable memory may also allow acoustic image datato be easily transferred to another computing device, or to be removedbefore acoustic analysis system 200 is used in another application.Processor 212 may also be implemented as a System on Chip thatintegrates some or all components of a computer or other electronicsystem into a single chip. The processor 212 (processing circuitry) canbe configured to communicate the processed data to a display 214 orother output/control device 218.

In some embodiments, acoustic sensors in acoustic sensor array 202generate a series of signals corresponding to the acoustic signalsreceived by each acoustic sensor to represent an acoustic image. A“frame” of acoustic image data is generated when the signal from eachacoustic sensor is obtained by scanning all of the rows that make up theacoustic sensor array 202. In some examples, processor 212 can acquireacoustic image frames at a rate sufficient to generate a videorepresentation (e.g. 30 Hz, or 60 Hz) of the acoustic image data.Independent of the specific circuitry, acoustic analysis system 200 maybe configured to manipulate acoustic data representative of the acousticprofile of a target scene so as to provide an output that can bedisplayed, stored, transmitted, or otherwise utilized by a user.

In some embodiments, the “back propagation” of received acoustic signalsin order to generate acoustic image data comprises analyzing thereceived signals at the plurality of acoustic sensors in the acousticsensor array 202, for example, via the processor. In various examples,performing the back propagation is a function of one or more parameters,including a distance to target, frequency, sound intensity (e.g., dBlevel) sensor array dimensions/configuration, including, for example,the spacing and arrangement of individual sensors within one or morearrays, etc. In some embodiments, such parameters can be pre-programmedinto the system, for example, in memory. For example, acoustic sensorarray 202 properties can be stored in memory, such as internal memory ormemory associated particularly with the acoustic sensor array 202. Otherparameters, such as a distance to target, can be received a variety ofways. For instance, in some examples, the acoustic analysis system 200includes a distance measuring tool 204 in communication with theprocessor 212. The distance measuring tool can be configured to providedistance information representative of the distance from the distancemeasuring tool 204 to a particular location in the target scene. Variousdistance measuring tools can include a laser distance meter or otherknown distance measuring devices, such as other optical or audiodistance measurement devices. Additionally or alternatively, a distancemeasuring tool can be configured to generate three-dimensional depthdata such that each portion of a target scene has an associateddistance-to-target value. Thus, in some examples, a distance to targetmeasurement as used herein can correspond to a distance to each locationwithin a target scene. Such three-dimensional depth data can begenerated, for example, via a plurality of imaging tools havingdifferent view of a target scene, or via other known distance scanningtools. In general, in various embodiments, a distance measuring tool canbe used to perform one or more distance measurement functions, includingbut not limited to: laser distance measurement, active sonic distancemeasurement, passive ultrasonic distance measurement, LIDAR distancemeasurement, RADAR distance measurement, millimeter wave distancemeasurement, and the like.

Distance information from the distance measuring tool 204 can be used inthe back propagation calculation. Additionally or alternatively, thesystem 200 can include a user interface 216 into which a user maymanually enter a distance to target parameter. For example, a user mayenter a distance to target value into the system 200 in the event thatthe distance to a component suspected of producing acoustic signals isknown or is difficult to measure with the distance measuring tool 204.

In the illustrated embodiment, acoustic analysis system 200 includes anelectromagnetic imaging tool 203 for generating image datarepresentative of a target scene. Exemplary electromagnetic imagingtools can be configured to receive electromagnetic radiation from atarget scene and generate electromagnetic image data representative ofthe received electromagnetic radiation. In some examples,electromagnetic imaging tool 203 can be configured to generateelectromagnetic image data representative of a particular range ofwavelengths within the electromagnetic spectrum, such as infraredradiation, visible light radiation, and ultraviolet radiation. Forinstance, in some embodiments, an electromagnetic timing tool 203 caninclude one or more camera modules configured to generate image datarepresentative of a particular range of wavelengths in theelectromagnetic spectrum such as, for example, a visible light cameramodule 206.

Visible light camera modules are generally well known. For examples,various visible light camera modules are included in smartphones andnumerous other devices. In some embodiments, visible light camera module206 may be configured to receive visible light energy from a targetscene and to focus the visible light energy on a visible light sensorfor generation of visible light energy data, e.g., that can be displayedin the form of a visible light image on display 214 and/or stored inmemory. Visible light camera module 206 can include any suitablecomponents for performing the functions attributed to the module herein.In the example of FIG. 2 , visible light camera module 206 isillustrated as including visible light lens assembly 208 and visiblelight sensor 210. In some such embodiments, visible light lens assembly208 includes at least one lens that takes visible light energy emittedby a target scene and focuses the visible light energy on visible lightsensor 210. Visible light sensor 210 can include a plurality of visiblelight sensor elements such as, e.g., CMOS detectors, CCD detectors, PINdiodes, avalanche photo diodes, or the like. Visible light sensor 210responds to the focused energy by generating an electrical signal thatcan be converted and displayed as a visible light image on display 214.In some examples, the visible light module 206 is configurable by auser, and can provide output, for example, to display 214, in a varietyof formats. Visible light camera module 206 may include compensationfunctionality for varying lighting or other operating conditions or userpreferences. The visible light camera module may provide a digitaloutput including image data, which may include data in a variety offormats (e.g., RGB, CYMK, YCbCr, etc.).

In operation of some exemplary visible light camera modules 206, opticalenergy received from a target scene may pass through visible light lensassembly 208 and be focused on visible light sensor 210. When theoptical energy impinges upon the visible light sensor elements ofvisible light sensor 210, photons within the photodetectors may bereleased and converted into a detection current. Processor 212 canprocess this detection current to form a visible light image of thetarget scene.

During use of acoustic analysis system 200, processor 212 can controlvisible light camera module 206 to generate visible light data from acaptured target scene for creating a visible light image. The visiblelight data may include luminosity data indicative of the color(s)associated with different portions of the captured target scene and/orthe magnitude of light associated with different portions of thecaptured target scene. Processor 212 can generate a “frame” of visiblelight image data by measuring the response of each visible light sensorelement of acoustic analysis system 200 a single time. By generating aframe of visible light data, processor 212 captures visible light imageof a target scene at a given point in time. Processor 212 may alsorepeatedly measure the response of each visible light sensor element ofacoustic analysis system 200 so as to generate a dynamic visible lightimage (e.g., a video representation) of a target scene. In someexamples, the visible light camera module 206 may include its owndedicated processor or other circuitry (e.g., ASIC) capable of operatingthe visible light camera module 206. In some such embodiments, thededicated processor is in communication with processor 212 for providingvisible light image data (e.g., RGB image data) to processor 212. Inalternative embodiments, a dedicated processor for the visible lightcamera module 206 may be integrated into processor 212.

With each sensor element of visible light camera module 206 functioningas a sensor pixel, processor 212 can generate a two-dimensional image orpicture representation of the visible light from a target scene bytranslating an electrical response of each sensor element into atime-multiplexed electrical signal that can be processed, e.g., forvisualization on display 214 and/or storage in memory.

Processor 212 may control display 214 to display at least a portion of avisible light image of a captured target scene. In some examples,processor 212 controls display 214 so that the electrical response ofeach sensor element of visible light camera module 206 is associatedwith a single pixel on display 214. In other examples, processor 212 mayincrease or decrease the resolution of a visible light image so thatthere are more or fewer pixels displayed on display 214 than there aresensor elements in visible light camera module 206. Processor 212 maycontrol display 214 to display an entire visible light image (e.g., allportions of a target scene captured by acoustic analysis system 200) orless than an entire visible light image (e.g., a lesser port of theentire target scene captured by acoustic analysis system 200).

In some embodiments, processor 212 may control display 214 toconcurrently display at least a portion of the visible light imagecaptured by acoustic analysis system 200 and at least a portion of anacoustic image generated via acoustic sensor array 202. Such aconcurrent display may be useful in that an operator may reference thefeatures displayed in the visible light image to help view sources ofacoustic signals concurrently displayed in the acoustic image. Invarious examples, processor 212 may control display 214 to display thevisible light image and the acoustic image in side-by-side arrangement,in a picture-in-picture arrangement, where one of the images surroundsthe other of the images, or any other suitable arrangement where thevisible light and the acoustic image are concurrently displayed.

For example, processor 212 may control display 214 to display thevisible light image and the acoustic image in a combined arrangement. Insuch an arrangement, for a pixel or set of pixels in the visible lightimage representative of a portion of the target scene, there exists acorresponding pixel or set of pixels in the acoustic image,representative of substantially the same portion of the target scene. Invarious embodiments, the size and/or resolution of the acoustic andvisible light images need not be the same. Accordingly, there may exista set of pixels in one of the acoustic or visible light images thatcorrespond to a single pixel in the other of the acoustic or visiblelight image, or a set of pixels of a different size. Similarly, theremay exist a pixel in one of the visible light or acoustic images thatcorresponds to a set of pixels in the other image. Thus, as used herein,corresponding does not require a one-to-one pixel relationship, but mayinclude mismatched sizes of pixels or groups of pixels. Variouscombination techniques of mismatched sized regions of images may beperformed, such as up- or down-sampling one of the images, or combininga pixel with the average value of a corresponding set of pixels. Otherexamples are known and are within the scope of this disclosure.

Thus, corresponding pixels need not have a direct one-to-onerelationship. Rather, in some embodiments, a single acoustic pixel has aplurality of corresponding visible light pixels, or a visible lightpixel has a plurality of corresponding acoustic pixels. Additionally oralternatively, in some embodiments, not all visible light pixels havecorresponding acoustic pixels, or vice versa. Such embodiments may beindicative of, for example, a picture-in-picture type display aspreviously discussed. Thus, a visible light pixel will not necessarilyhave the same pixel coordinate within the visible light image as does acorresponding acoustic pixel. Accordingly, as used herein, correspondingpixels generally refers pixels from any image (e.g., a visible lightimage, an acoustic image, a combined image, a display image, etc.)comprising information from substantially the same portion of the targetscene. Such pixels need not have a one-to-one relationship betweenimages and need not have similar coordinate positions within theirrespective images.

Similarly, images having corresponding pixels (i.e., pixelsrepresentative of the same portion of the target scene) can be referredto as corresponding images. Thus, in some such arrangements, thecorresponding visible light image and the acoustic image may besuperimposed on top of one another, at corresponding pixels. An operatormay interact with user interface 216 to control the transparency oropaqueness of one or both of the images displayed on display 214. Forexample, the operator may interact with user interface 216 to adjust theacoustic image between being completely transparent and completelyopaque and also adjust the visible light image between being completelytransparent and completely opaque. Such an exemplary combinedarrangement, which may be referred to as an alpha-blended arrangement,may allow an operator to adjust display 214 to display an acoustic-onlyimage, a visible light-only image, of any overlapping combination of thetwo images between the extremes of an acoustic-only image and a visiblelight-only image. Processor 212 may also combine scene information withother data, such as alarm data or the like. In general, an alpha-blendedcombination of visible light and acoustic images can comprise anywherefrom 100 percent acoustic and 0 percent visible light to 0 percentacoustic and 100 percent visible light. In some embodiments, the amountof blending can be adjusted by a user of the camera. Thus, in someembodiments, a blended image can be adjusted between 100 percent visiblelight and 100 percent acoustic.

Additionally, in some embodiments, the processor 212 can interpret andexecute commands from user interface 216, and/or output/control device218. Moreover, input signals may be used to alter the processing of thevisible light and/or acoustic image data that occurs in the processor212.

An operator may interact with acoustic analysis system 200 via userinterface 216, which may include buttons, keys, or another mechanism forreceiving input from a user. The operator may receive output fromacoustic analysis system 200 via display 214. Display 214 may beconfigured to display an acoustic-image and/or a visible light image inany acceptable palette, or color scheme, and the palette may vary, e.g.,in response to user control. In some embodiments, acoustic image datacan be presented in a palette in order to represent varying magnitudesof acoustic data from different locations in the scene. For instance, insome examples, display 214 is configured to display an acoustic image ina monochromatic palette such as grayscale. In other examples, display214 is configured to display an acoustic image in a color palette suchas, e.g., amber, ironbow, blue-red, or other high contrast color scheme.Combinations of grayscale and color palette displays are alsocontemplated. In some examples, the display being configured to displaysuch information may include processing capabilities for generating andpresenting such image data. In other examples, being configured todisplay such information may include the ability to receive image datafrom other components, such as processor 212. For example, processor 212may generate values (e.g., RGB values, grayscale values, or otherdisplay options) for each pixel to be displayed. Display 214 may receivesuch information and map each pixel into a visual display.

While processor 212 can control display 214 to concurrently display atleast a portion of an acoustic image and at least a portion of a visiblelight image in any suitable arrangement, a picture-in-picturearrangement may help an operator to easily focus and/or interpret anacoustic image by displaying a corresponding visible image of the samescene in adjacent alignment.

A power supply (not shown) delivers operating power to the variouscomponents of acoustic analysis system 200. In various examples, powersupply may include a rechargeable or non-rechargeable battery and apower generation circuit, AC power, an inductive power pick-up, aphotovoltaic power source, or any other appropriate power supplyingcomponent. Combinations of power supplying components are also possible,such as a rechargeable battery and another component configured toprovide power to operate the device and/or to charge the rechargeablebattery.

During operation of acoustic analysis system 200, processor 212 controlsacoustic sensor array 202 and visible light camera module 206 with theaid of instructions associated with program information that is storedin memory to generate a visible light image and an acoustic image of atarget scene. Processor 212 further controls display 214 to display thevisible light image and/or the acoustic image generated by acousticanalysis system 200.

As noted, in some situations, it can be difficult to identify anddifferentiate between real-world (visible) features of the target scenein an acoustic image. In addition to supplementing the acoustic imagewith visible light information, in some embodiments, it can be useful toemphasize visible edges within the target scene. In some embodiments,known edge detection methods can be performed on a visible light imageof a target scene. Because of the corresponding relationship between theacoustic image and the visible light image, visible light pixelsdetermined to represent a visible edge in the target scene correspond toacoustic pixels also representing the visible edge in the acousticimage. It will be appreciated that, as used herein, “edges” need notrefer to the physical boundary of an object, but may refer to anysufficiently sharp gradient in the visible light image. Examples mayinclude physical boundaries of an object, color changes within anobject, shadows across a scene, and the like.

While generally described with respect to FIG. 2 as including a visiblelight camera module 206, in some examples, electromagnetic imaging tool203 of acoustic analysis system 200 can additionally or alternativelyinclude imaging tools capable of generating image data representative ofa variety of spectrums. For instance, in various examples,electromagnetic imaging tool 203 can include one or more tools capableof generating infrared image data, visible light image data, ultravioletimage data, or any other useful wavelengths, or combinations thereof. Insome embodiments, the acoustic imaging system can include an infraredcamera module having an infrared lens assembly and an infrared sensorarray. Additional components for interfacing with, for example, aninfrared camera module can be included, such as those described in U.S.patent application Ser. No. 14/837,757, filed Aug. 27, 2015, andentitled “EDGE ENHANCEMENT FOR THERMAL-VISIBLE COMBINED IMAGES ANDCAMERAS,” which is assigned to the assignee of the instant applicationand is hereby incorporated by reference in its entirety.

In some examples, two or more data streams can be blended for display.For example, exemplary systems including a visible light camera module206, an acoustic sensor array 202, and an infrared camera module (notshown in FIG. 2 ) can be configured to produce an output imagecomprising a blend of visible light (VL) image data, infrared (IR) imagedata, and acoustic (Acoustic) image data. In an exemplary blendingscheme, the display image can be represented by: α×IR+β×VL+γ×Acoustic,wherein α+β+γ=1. In general, any number of data streams can be combinedfor display. In various embodiments, blending ratios such as α, β, and γcan be set by a user. Additionally or alternatively, set displayprograms can be configured to include different image data streams basedon alarm conditions (e.g., one or more values in one or more datastreams meets a predetermined threshold) or other conditions, forexample, as described in U.S. Pat. No. 7,538,326, entitled “VISIBLELIGHT AND IR COMBINED IMAGE CAMERA WITH A LASER POINTER,” which isassigned to the assignee of the instant application, and is herebyincorporated by reference in its entirety.

One of more components in acoustic analysis system 200 described withrespect to FIG. 2 can be included in a portable (e.g., handheld)acoustic analysis tool. For instance, in some embodiments, a portableacoustic analysis tool can include a housing 230 configured to house thecomponents in the acoustic analysis tool. In some examples, one or morecomponents of the system 200 can be located external to housing 230 ofan acoustic analysis tool. For instance, in some embodiments, processor212, display 214, user interface 216, and/or output control device 218can be located external to a housing of an acoustic analysis tool andcan communicate with various other system components, for example, viawireless communication (e.g., Bluetooth communication, Wi-Fi, etc.).Such components external to the acoustic analysis tool can be provided,for example, via an external device, such as a computer, smartphone,tablet, wearable device, or the like. Additionally or alternatively,other test and measurement or data acquisition tools configured to actas a master or slave device with respect to the acoustic analysis toolcan similarly provide various components of an acoustic analysis systemexternal to the acoustic analysis tool. External devices can communicatewith a portable acoustic analysis tool via a wired and/or wirelessconnection, and can be used to perform various processing, display,and/or interface steps.

In some embodiments, such external devices can provide redundantfunctionality as components housed in a portable acoustic analysis tool.For example, in some embodiments, an acoustic analysis tool can includea display for displaying acoustic image data and can further beconfigured to communicate image data to an external device for storageand/or display. Similarly, in some embodiments, a user may interfacewith an acoustic analysis tool via an application (an “app”) running ona smartphone, tablet, computer or the like, in order to perform one ormore functions also capable of being performed with the acousticanalysis tool itself.

FIG. 3A is a schematic diagram of an exemplary configuration of acousticsensor array within an acoustic analysis system. In the illustratedexample, the acoustic sensor array 302 includes a plurality of firstacoustic sensors (shown in white) and a plurality of second acousticsensors (shaded). The first acoustic sensors are arranged into a firstarray 320, and the second acoustic sensors are arranged into a secondarray 322. In some examples, the first array 320 and the second array322 can be selectively used to receive acoustic signals for generatingacoustic image data. For instance, in some configurations, thesensitivity of a particular acoustic sensor array to particular acousticfrequencies is a function of the distance between acoustic sensorelements.

In some configurations, more closely spaced together sensor elements(e.g., second array 322) are better able to resolve high frequencyacoustic signals (for example, sounds having frequencies greater than 20kHz, such as ultrasound signals between 20 kHz and 100 kHz) than furtherspaced sensor elements (e.g., first array 320). Similarly, furtherspaced sensor elements (e.g., first array 320) may be better suited fordetecting lower frequency acoustic signals (e.g., <20 kHz) than moreclosely spaced sensor elements (e.g., second array 322). Variousacoustic sensor arrays can be provided having sensor elements spacedapart from one another for detecting acoustic signals of variousfrequency ranges, such as infrasonic frequencies (<20 Hz), audiblefrequencies (between approximately 20 Hz and 20 kHz), ultrasoundfrequencies (between 20 kHz and 100 kHz). In some embodiments, partialarrays can be used (e.g., every other acoustic sensor element from array320) for optimizing detection of particular frequency bands.

Additionally, in some examples, some acoustic sensor elements may bebetter suited for detecting acoustic signals having different frequencycharacteristics, such as low or high frequencies. Thus, in someembodiments, an array configured for detecting low frequency acousticsignals, such as the first array 320 having further spaced sensorelements, may include first acoustic sensor elements better suited fordetecting low frequency acoustic signals. Similarly, an array configuredfor detecting higher frequency acoustic signals, such as second array322, may include second acoustic sensor elements better suited fordetecting high frequency acoustic signals. Thus, in some examples, thefirst array 320 and the second array 322 of acoustic sensor elements mayinclude different types of acoustic sensor elements. Alternatively, insome embodiments, the first array 320 and the second array 322 caninclude the same type of acoustic sensor element.

Thus, in an exemplary embodiment, an acoustic sensor array 302 caninclude a plurality of acoustic sensor element arrays, such as the firstarray 320 and the second array 322. In some embodiments, arrays can beused individually or in combination. For instance, in some examples, auser may select to use the first array 320, use the second array 322, oruse both the first array 320 and the second array 322 simultaneously forperforming an acoustic imaging procedure. In some examples, a user mayselect which array(s) are to be used via the user interface.Additionally or alternatively, in some embodiments, the acousticanalysis system may automatically select the array(s) to use based onanalysis of received acoustic signals or other input data, such as anexpected frequency range, or the like. While the configuration shown inFIG. 3A generally includes two arrays (first array 320 and second array322) generally arranged in rectangular lattices, it will be appreciatedthat a plurality of acoustic sensor elements can be grouped into anynumber of discrete arrays in any shape. Moreover, in some embodiments,one or more acoustic sensor elements can be included in multipledistinct arrays that can be selected for operation. As describedelsewhere herein, in various embodiments, processes for back propagatingacoustic signals to establish acoustic image data from the scene isperformed based on the arrangement of acoustic sensor elements. Thus,the arrangement of acoustic sensors may be known or otherwise accessibleby the processor in order to perform acoustic image generationtechniques.

The acoustic analysis system of FIG. 3A further includes a distancemeasuring tool 304 and a camera module 306 positioned within theacoustic sensor array 302. Camera module 306 can represent a cameramodule of an electromagnetic imaging tool (e.g., 203), and can include avisible light camera module, an infrared camera module, an ultravioletcamera module, or the like. Additionally, while not shown in FIG. 3A,the acoustic analysis system can include one or more additional cameramodules of the same type or of a different type than camera module 306.In the illustrated example, distance measuring tool 304 and cameramodule 306 are positioned within the lattice of acoustic sensor elementsin the first array 320 and the second array 322. While shown as beingdisposed between lattice sites within the first array 320 and the secondarray 322, in some embodiments, one or more components (e.g., cameramodule 306 and/or a distance measuring tool 304 can be positioned atcorresponding one or more lattice sites in the first array 320 and/orthe second array 322. In some such embodiments, the component(s) can bepositioned at the lattice site in lieu of an acoustic sensor elementthat would typically be in such a location according to the latticearrangement.

As described elsewhere herein, acoustic sensor arrays can includeacoustic sensor elements arranged in any of a variety of configurations.FIGS. 3B and 3C are schematic diagrams illustrating exemplary acousticsensor array configurations. FIG. 3B shows an acoustic sensor array 390that includes a plurality of acoustic sensor elements spaced evenlyapart in an approximately square lattice. Distance measuring tool 314and camera array 316 are positioned within acoustic sensor array 390. Inthe illustrated example, the acoustic sensor elements in acoustic sensorarray 390 are the same type of sensor, though in some embodiments,different types of acoustic sensor elements can be used in the array390.

FIG. 3C shows a plurality of acoustic sensor arrays. Acoustic sensorarrays 392, 394, and 396 each include a plurality of acoustic sensorelements arranged in a different shaped array. In the example of FIG.3C, acoustic sensor arrays 392, 394, and 396 can be used separately ortogether in any combination to create sensor arrays of various sizes. Inthe illustrated embodiment, the sensor elements of array 396 are spacedcloser together than the sensor elements of array 392. In some examples,array 396 is designed for sensing high frequency acoustic data, whilearray 392 is designed for sensing lower frequency acoustic data.

In various embodiments, arrays 392, 394, and 396 can include the same ordifferent types of acoustic sensor elements. For example, acousticsensor array 392 can include sensor elements having a frequencyoperating range lower than that of sensor elements of acoustic sensorarray 396.

As described elsewhere herein, in some examples, different acousticsensor arrays (e.g., 392, 394, 396) can be selectively turned off and onduring various modes of operation (e.g., different desired frequencyspectrums to be imaged). Additionally or alternatively, various acousticsensor elements (e.g., some or all of acoustic sensor elements in one ormore sensor arrays) can be enabled or disabled according to a desiredsystem operation. For example, in some acoustic imaging processes, whiledata from a large number of sensor elements (e.g., sensor elementsarranged in a high density, such as in sensor array 396) marginallyimproves acoustic image data resolution, it is at the expense ofrequired processing to extract acoustic image data from the datareceived at each sensor element. That is, in some examples, theincreased processing demand (e.g., in cost, processing time, powerconsumption, etc.) necessary for processing a large number of inputsignal (e.g., from a large number of acoustic sensor elements) comparesnegatively to any additional signal resolution provided by theadditional data streams. Thus, it may be worthwhile in some embodimentsto disable or disregard data from one or more acoustic sensor elementsdepending on the desired acoustic imaging operation.

Similar to the systems of FIGS. 3A and 3B, the system of FIG. 3Cincludes distance measuring tool 314 and camera array 316 positionedwithin acoustic sensor arrays 392, 394, and 396. In some examples,additional components, such as additional camera arrays (e.g., used forimaging a different portion of the electromagnetic spectrum from cameraarray 316), may be similarly positioned within acoustic sensor arrays392, 394, and 396. It will be appreciated that, while shown in FIGS.3A-2C as being positioned within one or more acoustic sensor arrays,distance measuring tool and/or one or more imaging tools (e.g., visiblelight camera module, infrared camera module, ultraviolet sensor, etc.)can be located outside of the acoustic sensor array(s). In some suchexamples, the distance measuring tool and/or one or more imaging toolslocated outside of the acoustic sensor array(s) can be supported by anacoustic imaging tool, for example, by a housing that houses theacoustic sensor array(s), or can located externally to the housing ofthe acoustic imaging tool.

In some examples, general misalignment of an acoustic sensor array andan imaging tool, such as a camera module, can lead to misalignment incorresponding image data generated by the acoustic sensor array and theimaging tool. FIG. 4A shows a schematic illustration of parallax errorin the generation of a frame of visible light image data and acousticimage data. In general, parallax error can be vertical, horizontal, orboth. In the illustrated embodiment, an acoustic sensor array 420 and animaging tool comprising a visible light camera module 406. Visible lightimage frame 440 is shown being captured according to the field of view441 of the visible light camera module 406 while acoustic image frame450 is shown as being captured according to the field of view 451 of theacoustic sensor array 420.

As shown, the visible light image frame 440 and the acoustic imagingframe 450 are not aligned with one another. In some embodiments, aprocessor (e.g., processor 212 of FIG. 2 ) is configured to manipulateone or both of the visible light image frame 440 and the acoustic imageframe 450 in order to align visible light image data and acoustic imagedata. Such manipulation can include shifting one image frame relative tothe other. The amount that the image frames are shifted relative to oneanother can be determined based on a variety of factors, including, forinstance, the distance to the target from the visible light cameramodule 406 and/or the acoustic sensor array 420. Such distance data canbe determined, for example, using distance measuring tool 404 orreceiving a distance value via a user interface (e.g., 216).

FIG. 4B is a schematic illustration similar to that in FIG. 4A, butincluding a visible light image of a scene. In the example of FIG. 4B,visible light image 442 shows a scene of a plurality of power lines anda supporting tower. The acoustic image 452 includes a plurality oflocations 454, 456, 458 indicating high magnitude acoustic data comingfrom such locations. As shown, the visible light image 442 and theacoustic image 452 are both displayed simultaneously. However,observation of both images shows at least one acoustic image localmaximum at location 458 that does not appear to coincide with anyparticular structure in the visible light image 442. Thus, one observingboth images may conclude that there is misalignment (e.g., parallaxerror) between the acoustic image 452 and the visible light image 442.

FIGS. 5A and 5B show parallax correction between a visible light imageand an acoustic image. FIG. 5A, similar to FIG. 4B, shows a visiblelight image 542 and an acoustic image 552. The acoustic image 552includes local maxima at locations 554, 556, and 558. As can be seen,maxima at locations 554 and 558 do not appear to coincide with anystructure in the visible light image. In the example of FIG. 5B, thevisible light image 542 and the acoustic image 552 are registered withrespect to one another. The local maxima at locations 554, 556, and 558in the acoustic image now appear to coincide with various locationswithin the visible light image 542.

During use, an operator may view the representation in FIG. 5B (e.g.,via display 214) and determine approximate locations in the visiblescene 542 that are likely sources of received acoustic signals. Suchsignals can be further processed in order to determine informationregarding the acoustic signature of various components in the scene. Invarious embodiments, acoustic parameters such as frequency content,periodicity, amplitude, and the like can be analyzed with respect tovarious locations in the acoustic image. When overlaid onto visiblelight data so that such parameters can be associated with various systemcomponents, acoustic image data may be used to analyze variousproperties (e.g., performance characteristics) of objects in the visiblelight image.

FIGS. 5C and 5D are colorized versions of FIGS. 5A and 5B. As shown inFIGS. 5A and 5B, and more easily seen in the colorized representation ofFIGS. 5C and 5D, locations 554, 556, and 558 show a circular gradient incolor. As described elsewhere herein, acoustic image data can bevisually represented according to a palettization scheme in which eachpixel of acoustic image data is colorized based on the acousticintensity at a corresponding location. Thus, in the exemplaryrepresentations of FIGS. 5A-5D, the circular gradient of locations 554,556, 558 generally represents a gradient in acoustic intensity in theimaging plane based on back-propagated received acoustic signals.

It will be appreciated that, while the exemplary illustrations in FIGS.4A, 4B, 5A-5D are described with respect to acoustic image data andvisible light image data, such processes can be similarly performed witha variety of electromagnetic image data. For example, as describedelsewhere herein, in various embodiments, various such processes can beperformed using combinations of acoustic image data and one or more ofvisible light image data, infrared image data, ultraviolet image data,or the like.

As described elsewhere herein, in some embodiments, the back-propagationof acoustic signals to form an acoustic image can be based on a distanceto target value. That is, in some examples, the back-propagationcalculations can be based on a distance, and can include determining atwo-dimensional acoustic scene located at that distance from theacoustic sensor array. Given a two-dimensional imaging plane, sphericalsound waves emanating from a source in the plane would generally appearcircular in cross-section, with a radial decay in intensity as shown inFIGS. 5A-5B.

In some such examples, portions of an acoustic scene representing datanot located at the distance-to-target used in the back-propagationcalculation will result in errors in the acoustic image data, such asinaccuracies in the location of one or more sounds in the scene. Sucherrors can, when the acoustic image is displayed simultaneously (e.g.,blended, combined, etc.) with other image data (e.g., electromagneticimage data, such as visible light, infrared, or ultraviolet image data),lead to parallax errors between the acoustic image data and other imagedata. Thus, in some embodiments, some techniques for correcting parallaxerror (e.g., as shown in FIGS. 5A and 5B) comprise adjusting adistance-to-target value used in the back-propagation calculation forgenerating acoustic image data.

In some cases, the system can be configured to perform aback-propagation process using a first distance-to-target value anddisplay a display image such as shown in FIG. 5A in which the acousticimage data and another data stream may not be aligned. Subsequently, theacoustic analysis system can adjust the distance-to-target value usedfor back-propagation, perform the back-propagation again, and update thedisplay image with the new acoustic image data. This process can repeat,with the acoustic analysis system cycling through a plurality ofdistance-to-target values while a user observes the resulting displayimages on a display. As the distance-to-target value changes, the usermay observe a gradual transition from the display image shown in FIG. 5Ato the display image shown in FIG. 5B. In some such cases, a user mayvisually observe when the acoustic image data appears properlyregistered with another data stream, such as electromagnetic image data.A user may signal to the acoustic analysis system that the acousticimage data appears properly registered, indicating to the system thatthe distance-to-target value used to perform the most recentback-propagation is approximately correct, and can save that distancevalue to memory as the correct distance-to-target. Similarly, a user maymanually adjust a distance-to-target value as the display image isupdated using new distance values in updated back-propagation processesuntil the user observes that the acoustic image data is properlyregistered. The user may choose to save the current distance to targetin the acoustic analysis system as a current distance-to-target.

In some examples, correcting a parallax error can include adjusting theposition of the acoustic image data relative to other image data (e.g.,electromagnetic image data) by a predetermined amount and in apredetermined direction based on the distance-to-target data. In someembodiments, such adjustments are independent of the generation of theacoustic image data by back-propagating acoustic signals to theidentified distance-to-target.

In some embodiments, in addition to being used to generate acousticimage data and reduce parallax error between the acoustic image data andother image data, a distance-to-target value can be used for otherdeterminations. For instance, in some examples, a processor (e.g., 212)can use a distance to target value in order to focus or assist a user infocusing an image, such as an infrared image, as described in U.S. Pat.No. 7,538,326, which is incorporated by reference. As described therein,this can similarly be used to correct for parallax errors betweenvisible light image data and infrared image data. Thus, in someexamples, a distance value can be used to register acoustic image datawith electromagnetic imaging data, such as infrared image data andvisible light image data.

As described elsewhere herein, in some examples, a distance measuringtool (e.g., 204) is configured to provide distance information that canbe used by the processor (e.g., 212) for generating and registeringacoustic image data. In some embodiments, the distance measuring toolcomprises a laser distance meter configured to emit light onto thetarget scene at a location to which the distance is measured. In somesuch examples, the laser distance meter can emit light in the visiblespectrum so that the user may view the laser spot in the physical sceneto ensure that the distance meter is measuring a distance to a desiredportion of the scene. Additionally or alternatively, the laser distancemeter is configured to emit light in a spectrum to which one or moreimaging components (e.g., camera modules) is sensitive. Thus, a userviewing the target scene via the analysis tool (e.g., via display 214)may observe the laser spot in the scene to ensure that the laser ismeasuring the distance to the correct location in the target scene. Insome examples, the processor (e.g., 212) can be configured to generate areference mark in a displayed image representative of the location thatthe laser spot would be located in the acoustic scene based on a currentdistance value (e.g., based on a known distance-based parallaxrelationship between the laser distance meter and the acoustic sensorarray). The reference mark location can be compared to a location of theactual laser mark (e.g., graphically on a display and/or physically inthe target scene) and the scene can be adjusted until the reference markand the laser coincide. Such processes can be performed similar to theinfrared registration and focusing techniques described in U.S. Pat. No.7,538,326, which is incorporated by reference.

FIG. 6 is a process flow diagram showing an exemplary method forgenerating a final image combining acoustic image data andelectromagnetic image data. The method includes the steps of receivingacoustic signals via an acoustic sensor array (680) and receivingdistance information (682). Distance information can be received, forexample, via a distance measuring device and/or a user interface, suchas via manual entry or as the result of a distance adjustment process bywhich the distance is determined based on an observed registration.

The method further includes back-propagating the received acousticsignals to determine acoustic image data representative of the acousticscene (684). As described elsewhere herein, back-propagating can beinclude analyzing a plurality of acoustic signals received at aplurality of sensor elements in an acoustic sensor array in combinationwith the received distance information to determine a source pattern ofthe received acoustic signals.

The method of FIG. 6 further includes the step of capturingelectromagnetic image data (686), and registering the acoustic imagedata with the electromagnetic image data (688). In some embodiments,registering the acoustic image data with the electromagnetic image datais done as part of the back-propagation step for generating the acousticimage data (684). In other examples, registering the acoustic image datawith the electromagnetic image data is done separately from thegeneration of the acoustic image data.

The method of FIG. 6 includes the step of combining acoustic image datawith electromagnetic image data in order to generate a display image(690). As described elsewhere herein, combining electromagnetic imagedata and acoustic image data can comprise alpha blending theelectromagnetic image data and the acoustic image data. Combining theimage data can include overlaying one image data set onto the other,such as in a picture-in-picture mode or in locations in which certainconditions (e.g., alarm conditions) are satisfied. The display image canbe presented to a user, for example, via a display that is supported bya housing that supports the acoustic sensor array and/or via a displayseparate from the sensor array, such as the display of an externaldevice (e.g., a smartphone, tablet, computer, etc.).

Additionally or alternatively, the display image can be saved in a local(e.g., on-board) memory and/or a remote memory for future viewing. Insome embodiments, the saved display image can include metadata thatallows for future adjustment of the display image properties, such asblending ratios, back-propagation distance, or other parameters used togenerate the image. In some examples, raw acoustic signal data and/orelectromagnetic image data can be saved with the display image forsubsequent processing or analysis.

While shown as a method for generating a final image combining acousticimage data and electromagnetic image data, it will be appreciated thatthe method of FIG. 6 could be used to combine acoustic image data withone or more sets of image data spanning any portion of theelectromagnetic spectrum, such as visible light image data, infraredimage data, ultraviolet image data, and the like. In some such examples,multiple sets of image data, such as visible light image data andinfrared image data, can be both combined with acoustic image data togenerate a display image via methods similar to that described withrespect to FIG. 6 .

In some examples, receiving acoustic signals via a sensor array (680)can include a step of selecting an acoustic sensor array with which toreceive acoustic signals. As described, for example, with respect toFIGS. 3A-C, acoustic analysis systems can include a plurality ofacoustic sensor arrays that can be suited for analyzing acoustic signalsof varying frequencies. Additionally or alternatively, in some examples,different acoustic sensor arrays can be useful for analyzing acousticsignals propagating from different distances. In some embodiments,different arrays can be nested inside one another. Additionally oralternatively, partial arrays can be selectively used to receiveacoustic image signals.

For example, FIG. 3A shows first array 320 and a second array 322 nestedwithin the first array. In an exemplary embodiment, the first array 320can include a sensor array configured (e.g., spaced) for receivingacoustic signals and generating acoustic image data for frequencies in afirst frequency range. Second array 322 can include, for example, asecond sensor array configured to be used alone or in combination withall or part of the first array 320 for generating acoustic image datafor frequencies in a second frequency range.

Similarly, FIG. 3C shows a first array 392, a second array 394 at leastpartially nested within the first array 392, and a third array 396 atleast partially nested within the first array 392 and the second array394. In some embodiments, the first array 392 can be configured forreceiving acoustic signals and generating acoustic image data forfrequencies in a first frequency range. The second array 394 can be usedwith all or part of the first array 392 for receiving acoustic signalsand generating acoustic image data for frequencies in a second frequencyrange. The third array 396 can be used alone, with all or part of thesecond array 394, and/or with all or part of the first array 392 forreceiving acoustic signals and generating acoustic image data forfrequencies in a third frequency range.

In some embodiments, in a nested array configuration, acoustic sensorelements from one array may be positioned between the acoustic sensorelements, such as elements of third array 396 being generally betweenelements of first array 392. In some such examples, the acoustic sensorelements in a nested array (e.g., third array 396) may be positioned inthe same plane as, in front of, or behind the acoustic sensor elementsin the array into which it is nested (e.g., first array 392).

In various implementations, arrays used for sensing higher frequencyacoustic signals generally require less distance between individualsensors. Thus, with respect to FIG. 3C, for instance, third array 396may be better suited for performing acoustic imaging processes involvinghigh frequency acoustic signals. Other sensor arrays (e.g., first array392) may be sufficient for performing acoustic imaging processesinvolving lower frequency signals, and may be used in order to reducethe computational demand of processing signals from a smaller number ofacoustic sensor elements when compared to array 396. Thus, in someexamples, high-frequency sensor arrays may be nested withinlow-frequency sensor arrays. As described elsewhere herein, such arraysmay generally be operated individually (e.g., via switching betweenactive arrays) or together.

In addition or alternatively to selecting an appropriate sensor arraybased on an expected/desired frequency spectrum for analysis, in someexamples, different sensor arrays may be better suited for performingacoustic imaging processes at difference distances to the target scene.For example, in some embodiments, if the distance between an acousticsensor array and a target scene is small, outer sensor elements in theacoustic sensor array may receive significantly less useful acousticinformation from the target scene than sensor elements more centrallylocated.

On the other hand, if the distance between an acoustic sensor array anda target scene is large, closely spaced acoustic sensor elements may notprovide separately useful information. That is, if first and secondacoustic sensor elements are close together, and the target scene isgenerally far away, the second acoustic sensor element may not provideany information that is meaningfully different from the first. Thus,data streams from such first and second sensor elements may be redundantand unnecessarily consume processing time and resources for analysis.

In addition to impacting which sensor arrays may be best suited forperforming acoustic imaging, as described elsewhere herein, the distanceto target may also be used in performing the back-propagating fordetermining acoustic image data from received acoustic signals. However,in addition to being an input value into a back-propagation algorithm,the distance-to-target may be used to select an appropriateback-propagation algorithm to use. For instance, in some examples, atfar distances, spherically-propagating sound waves may be approximatedas being substantially planar compared to the size of an acoustic sensorarray. Thus, in some embodiments, when the distance-to-target is large,back-propagation of received acoustic signals can include an acousticbeamforming calculation. However, when closer to the source of the soundwaves, a planar approximation of the sound wave may not be appropriate.Thus, different back-propagation algorithms may be used, such asnear-field acoustic holography.

As described, a distance-to-target metric can be used in a variety ofways in an acoustic imaging process, such as determining active sensorarray(s), determining a back-propagation algorithm, performing theback-propagation algorithm, and/or registering a resulting acousticimage with electromagnetic image data (e.g., visible light, infrared,etc.). FIG. 7 is a process-flow diagram showing an exemplary process forgenerating acoustic image data from received acoustic signals.

The process of FIG. 7 includes receiving distance information (780), forexample, from a distance measuring device or entered distanceinformation, such as via a user interface. The method further includesthe step of selecting one or more acoustic sensor array(s) forperforming acoustic imaging based on the received distance information(782). As described, in various examples, selected array(s) can includea single array, a combination of multiple arrays, or portions of one ormore arrays.

The method of FIG. 7 further includes the step of selecting a processingscheme for performing acoustic imaging based on the received distanceinformation (784). In some examples, selecting a processing scheme caninclude selecting a back-propagation algorithm for generating acousticimage data from acoustic signals.

After selecting an acoustic sensor array (782) and processing scheme(784) for performing acoustic imaging, the method includes the steps ofreceiving acoustic signals via the selected acoustic sensor array (786).The received acoustic signals are then back-propagated using thedistance and the selected processing scheme to determine acoustic imagedata (788).

In various embodiments, steps of FIG. 7 can be performed by a user, anacoustic analysis system (e.g., via processor 212), or a combinationthereof. For example, in some embodiments, a processor can be configuredto receive distance information (780) via a distance measuring tooland/or a user input. In some examples, a user can input a value tooverride a measured distance to use as the distance information, forinstance, if the distance to an object is known and/or is difficult toanalyze via the distance measuring tool (e.g., small object size and/orlarge distance to target, etc.). The processor can be further configuredto automatically select an appropriate acoustic sensor array forperforming acoustic imaging based on the received distance information,for example, using a lookup table or other database. In someembodiments, selecting an acoustic sensor array comprises enablingand/or disabling one or more acoustic sensor elements in order toachieve a desired acoustic sensor array.

Similarly, in some examples, the processor can be configured toautomatically select a processing scheme (e.g., back-propagationalgorithm) for performing acoustic imaging based on the receiveddistance information. In some such examples, this can include selectingone from a plurality of known processing schemes stored in memory.Additionally or alternatively, selecting a processing scheme may amountto adjusting portions of a single algorithm to arrive at a desiredprocessing scheme. For example, in some embodiments, a singleback-propagation algorithm may include a plurality of terms and variable(e.g., based on distance information). In some such examples, selectinga processing scheme (784) can include defining one or more values in thesingle algorithm, such as adjusting coefficients for one or more terms(e.g., setting various coefficients to zero or one, etc.).

Thus, in some embodiments, an acoustic imaging system can automateseveral steps of an acoustic imaging process by suggesting and/orautomatically implementing a selected acoustic sensor array and/or aprocessing scheme (e.g., a back-propagation algorithm) based on receiveddistance data. This can speed up, improve, and simplify acoustic imagingprocesses, eliminating the requirements of an acoustic imaging expert tocarry out an acoustic imaging process. Thus, in various examples, theacoustic imaging system can automatically implement such parameters,notify the user that such parameters are about to implemented, ask auser for permission to implement such parameters, suggest suchparameters for manual input by a user, or the like.

Automatic selection and/or suggestion of such parameters (e.g.,processing scheme, sensor array) can be useful to optimize localizationof the acoustic image data with respect to other forms of image data,processing speed, and analysis of the acoustic image data. For instance,as described elsewhere herein, accurate back-propagation determination(e.g., using a proper algorithm and/or an accurate distance metric) canreduce parallax errors between acoustic image data and other (e.g.,electromagnetic, such as visible light, infrared, etc.) image data.Additionally, utilizing proper algorithms and/or sensor arrays such asmay be automatically selected or suggested by an acoustic analysissystem can optimize the accuracy of the thermal image data, allowing foranalysis of the received acoustic data.

As described, in some examples, an acoustic analysis system can beconfigured to automatically select an algorithm and/or a sensor arrayfor performing acoustic imaging processes based on received distanceinformation. In some such embodiments, a system includes a lookup table,for example, stored in memory, for determining which of a plurality ofback-propagation algorithms and acoustic sensor arrays to use fordetermining acoustic image data. FIG. 8 shows an exemplary lookup tablefor determining an appropriate algorithm and sensor array for use duringan acoustic imaging process.

In the illustrated example, the lookup table of FIG. 8 includes Ncolumns, each representing different arrays: Array 1, Array 2, . . . ,Array N. In various examples, each array comprises a unique set ofacoustic sensor elements arranged. Different arrays may comprise sensorelements arranged into a lattice (e.g., array 392 and array 396 in FIG.3C). Arrays within the lookup table can also include combinations ofsensor elements from one or more such lattices. In general, in someembodiments, each of arrays Array 1, Array 2, . . . , Array Ncorresponds to a unique combination of acoustic sensor elements. Some ofsuch combinations can include an entire set of sensor elements arrangedin a particular lattice, or can include a subset of sensor elementsarranged in a particular lattice. Any of a variety of combinations ofacoustic sensor elements are possible options for use as a sensor arrayin the lookup table.

The lookup table of FIG. 8 further includes M rows, each representing adifferent algorithm: Algorithm 1, Algorithm 2, . . . , Algorithm M. Insome examples, different algorithms may include different processes forperforming back-propagation analysis of received acoustic signals. Asdescribed elsewhere herein, in some examples, some different algorithmscan be similar to one another while having different coefficients and/orterms for modifying the back-propagation results.

The exemplary lookup table of FIG. 8 includes M×N entries. In someembodiments, the acoustic analysis system utilizing such a lookup tableis configured to analyze received distance information and classify thedistance information into one of M×N bins, wherein each bin correspondsto an entry in the lookup table of FIG. 8 . In such examples, when theacoustic analysis system receives distance information, the system canfind the entry (i, j) in the lookup table corresponding to the bin inwhich the distance information lies, and determine the appropriatealgorithm and sensor array for use during an acoustic imaging process.For example, if the received distance information corresponds to the binassociated with entry (i, j), the acoustic analysis system canautomatically utilize or suggest using Algorithm i and Array j for anacoustic imaging process.

In various such examples, distance information bins can correspond touniformly sized distance ranges, for example, a first bin correspondingto distances of within one foot, a second bin corresponding to distancesbetween one and two feet, and so on. In other examples, bins need notcorrespond to uniformly sized distance spans. Additionally, in someembodiments, fewer than M×N bins can be used. For example, in someembodiments, there may be an algorithm (e.g., Algorithm x) that is notever used with a particular array (e.g., Array y). Thus, in such anexample, there would be no corresponding distance information bincorresponding to entry (x, y) in the M×N lookup table.

In some embodiments, statistical analysis on the populated distance binscan be used for identifying a most common distance or distance rangewithin the target scene. In some such embodiments, the distance binhaving the highest number of corresponding locations (e.g., a highestnumber of locations with acoustic signals) can be used as distanceinformation in the process of FIG. 7 . That is, in some embodiments, autilized acoustic sensor array and/or processing scheme may beimplemented and/or recommended based on statistical analysis of thedistance distribution of various objects in the target scene. This canincrease the likelihood that sensor array and/or processing scheme usedfor acoustic imaging of a scene is appropriate for the largest number oflocations within the acoustic scene.

Additionally or alternatively, parameters other than distanceinformation can be used to select appropriate sensor arrays and/orprocessing schemes to use in generating acoustic image data. Asdescribed elsewhere herein, various sensor arrays can be configured tobe sensitive to certain frequencies and/or frequency bands. In someexamples, different back-propagation calculations similar can be usedaccording to different acoustic signal frequency content. Thus, in someexamples, one or more parameters can be used to determine a processingscheme and/or acoustic sensor array.

In some embodiments, the acoustic analysis system can be used toinitially analyze various parameters of received acoustic signalsprocessing/analysis. With reference back to FIG. 7 , a method forgenerating acoustic image data can include a step of, after receivingacoustic signals (786), analyzing frequency content of the receivedsignals (790). In some such examples, if the acoustic sensor array(s)and/or processing schemes have been selected (e.g., via steps 782 and/or784, respectively), the method can include the steps of updating theselected array(s) and/or updating the selected processing scheme (792),for example, based on the analyzed frequency content.

After updating the sensor array(s) and/or processing scheme, the methodcan perform various actions using the updated parameters. For example,if the selected sensor array(s) is updated (792) based on analyzedfrequency content (790), then new acoustic signals can be received fromthe (newly) selected acoustic sensor array (786), which can then beback-propagated to determine acoustic image data (788). Alternatively,if the processing scheme is updated at 792, already-captured acousticsignals can be back-propagated according to the updated processingscheme to determine updated acoustic image data. If both the processingscheme and the sensor array(s) are updated, new acoustic signals can bereceived using the updated sensor array and can be back-propagatedaccording to the updated processing scheme.

In some embodiments, an acoustic analysis system can receive frequencyinformation (778) without analyzing frequency content of receivedacoustic signals (790). For instance, in some examples, an acousticanalysis system can receive information regarding a desired or expectedfrequency range for future acoustic analysis. In some such examples, thedesired or expected frequency information can be used to select one ormore sensor arrays and/or a processing scheme that best fits thefrequency information. In some such examples, the step(s) of selectingacoustic sensor array(s) (782) and/or selecting a processing scheme(784) can be based on received frequency information in addition oralternatively to received distance information.

In some examples, received acoustic signals (e.g., received via theacoustic sensor elements) can be analyzed, for example, via a processor(e.g., 210) of an acoustic analysis system. Such analysis can be used todetermine one or more properties of the acoustic signals, such asfrequency, intensity, periodicity, apparent proximity (e.g., a distanceestimated based on received acoustic signals), measured proximity, orany combinations thereof. In some examples, acoustic image data can befiltered, for instance, to only show acoustic image data representingacoustic signals having a particular frequency content, periodicity, orthe like. In some examples, any number of such filters can be appliedsimultaneously.

As described elsewhere herein, in some embodiments, a series of framesof acoustic image data can be captured over time, similar to acousticvideo data. Additionally or alternatively, even if acoustic image datais not repeatedly generated, in some examples, acoustic signals arerepeatedly sampled and analyzed. Thus, with or without repeated acousticimage data generation (e.g., video), parameters of acoustic data, suchas frequency, can be monitored over time.

FIG. 9A is an exemplary plot of frequency content of received image dataover time in an acoustic scene. As shown, the acoustic scene representedby the plot of FIG. 9A generally includes four sustained frequenciesover time, labeled as Frequency 1, Frequency 2, Frequency 3, andFrequency 4. Frequency data, such as the frequency content of a targetscene, can be determined via processing received acoustic signals, forexample, using a Fast Fourier Transform (FFT) or other known method offrequency analysis.

FIG. 9B shows an exemplary scene including a plurality of locationsemitting acoustic signals. In the illustrated image, acoustic image datais combined with visible light image data, and shows acoustic signalspresent at locations 910, 920, 930, and 940. In some embodiments,acoustic analysis system is configured to display acoustic image data ofany detected frequency range. For instance, in an exemplary embodiment,location 910 includes acoustic image data including Frequency 1,location 920 includes acoustic image data including Frequency 2,location 930 includes acoustic image data including Frequency 3, andlocation 940 includes acoustic image data including Frequency 4.

In some such examples, displaying acoustic image data representativefrequency ranges is a selectable mode of operation. Similarly, in someembodiments, acoustic analysis system is configures to display acousticimage data representative of frequencies only within a predeterminedfrequency band. In some such examples, displaying acoustic image datarepresenting a predetermined frequency range comprises selecting one ormore acoustic sensor arrays for receiving acoustic signals from which togenerate acoustic image data. Such arrays can be configured to receive aselective frequency range. Similarly, in some examples, one or morefilters can be employed to limit the frequency content used to generatethe acoustic image data. Additionally or alternatively, in someembodiments, acoustic image data comprising information representativeof a broad range of frequencies can be analyzed and shown on the displayonly if the acoustic image data satisfies a predetermined condition(e.g., falls within a predetermined frequency range).

FIG. 9C shows a plurality of combined acoustic and visible light imagedata at a plurality of predefined frequency ranges. A first imageincludes acoustic image data at a first location 910 that includesfrequency content of Frequency 1. A second image includes acoustic imagedata at a second location 920 that includes frequency content ofFrequency 2. A third image includes acoustic image data at a thirdlocation 930 that includes frequency content of Frequency 3. A fourthimage includes acoustic image data at a fourth location 940 thatincludes frequency content of Frequency 4.

In an exemplary embodiment, a user may select various frequency ranges,such as ranges including Frequency 1, Frequency 2, Frequency 3, orFrequency 4, for filtering acoustic image data representative offrequency content other than the selected frequency range. Thus, in suchexamples, any of the first, second, third, or fourth images may bedisplayed as a result of a desired frequency range being selected by auser.

Additionally or alternatively, in some examples, an acoustic analysissystem may cycle between a plurality of display images, each havingdifferent frequency content. For instance, with respect to FIG. 9C, inan exemplary embodiment, an acoustic analysis system may display, in asequence, the first, second, third, and fourth images, such as shown bythe arrows in FIG. 9C.

In some examples, display images can includes a text or other displayrepresentative of the frequency content being displayed in the image sothat a user may observe which locations in the image include acousticimage data representative of certain frequency content. For example,with respect to FIG. 9C, each image may show a textual representation ofthe frequency represented in the acoustic image data. With respect toFIG. 9B, an image showing a plurality of frequency ranges may includeindications of the frequency content at each location including acousticimage data. In some such examples, a user may select a location in theimage, for example, via a user interface, for which to view thefrequency content present at that location it the acoustic scene. Forexample, a user may select first location 910, and the acoustic analysissystem may present the frequency content of the first location (e.g.,Frequency 1). Thus, in various examples, a user can use the acousticanalysis system in order to analyze the frequency content of an acousticscene, such as by viewing where in the scene corresponds to certainfrequency content and/or by viewing what frequency content is present atvarious locations.

During exemplary acoustic imaging operation, filtering acoustic imagedata by frequency can help reduce image clutter, for example, frombackground or other unimportant sounds. In an exemplary acoustic imagingprocedure, a user may wish to eliminate background sounds, such as floornoise in an industrial setting. In some such instances, background noisecan include mostly low frequency noise. Thus, a user may choose to showacoustic image data representative of acoustic signals greater than apredetermined frequency (e.g., 10 kHz). In another example, a user maywish to analyze a particular object that generally emits acousticsignals within a certain range, such as corona discharge from atransmission line (e.g., as shown in FIGS. 5A-D5). In such an example, auser may select a particular frequency range (e.g., between 11 kHz and14 kHz for corona discharge) for acoustic imaging.

In some examples, an acoustic analysis system can be used to analyzeand/or present information associated with the intensity of receivedacoustic signals. For example, in some embodiments, back-propagatingreceived acoustic signals can include determining an acoustic intensityvalue at a plurality of locations in the acoustic scene. In someexamples, similar to frequency described above, acoustic image data isonly included in a display image if the intensity of the acousticsignals meets one or more predetermined requirements.

In various such embodiments, a display image can include acoustic imagedata representative of acoustic signals above a predetermined threshold(e.g., 15 dB), acoustic signals below a predetermined threshold (e.g.,100 dB), or acoustic signals within a predetermined intensity range(e.g., between 15 dB and 40 dB). In some embodiments, a threshold valuecan be based on a statistical analysis of the acoustic scene, such asabove or below a standard deviation from the mean acoustic intensity.

Similar to as described above with respect to frequency information, insome embodiments, restricting acoustic image data to represent acousticsignals satisfying one or more intensity requirements can includefiltering received acoustic signals so that only received signals thatsatisfy the predetermined conditions are used to generate acoustic imagedata. In other examples, acoustic image data is filtered to adjust whichacoustic image data is displayed.

Additionally or alternatively, in some embodiments, acoustic intensityat locations within an acoustic scene can be monitored over time (e.g.,in conjunction with a video acoustic image representation or viabackground analysis without necessarily updating a display image). Insome such examples, predetermined requirements for displaying acousticimage data can include an amount or rate of change in acoustic intensityat a location in an image.

FIGS. 10A and 10B are exemplary display images including combinedvisible light image data and acoustic image data. FIG. 10A shows adisplay image including acoustic image data shown at a plurality oflocations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. Insome examples, intensity values can be palettized, for example, whereinan acoustic intensity value is assigned a color based on a predeterminedpalettization scheme. In an exemplary embodiment, intensity values canbe categorized according to intensity ranges (e.g., 10 dB-20 dB, 20dB-30 dB, etc.). Each intensity range can be associated with aparticular color according to a palettization scheme. Acoustic imagedata can include a plurality of pixels, wherein each pixel is colorizedin the color associated with the intensity range into which theintensity represented by the pixel of acoustic image data falls. Inaddition or alternatively to being differentiated by color, differentintensities can be distinguished according to other properties, such astransparency (e.g., in an image overlay in which acoustic image data isoverlaid onto other image data) or the like.

Additional parameters may also be palettized, such as a rate of changeof acoustic intensity. Similar to intensity, varying rates of change inacoustic intensity can be palettized such that portions of the scenesexhibiting different rates and/or amounts of acoustic intensity changeare displayed in different colors.

In the illustrated example, the acoustic image data is palettizedaccording to an intensity palette, such that acoustic image datarepresentative of different acoustic signal intensities are shown in adifferent color and/or shade. For instance, acoustic image data atlocations 1010 and 1030 show a palletized representation of a firstintensity, locations 1040, 1060, and 1080 show a palletizedrepresentation of a second intensity, and locations 1020, 1050, 1070,and 1090 show a palletized representation of a third intensity. As shownin the exemplary representation in FIG. 10A, each location showing apalettized representation of acoustic image data shows circular patternhaving a color gradient extending outward from the center. This can bedue to the decay of acoustic intensity as the signals propagate from asource of the acoustic signals.

In the example of FIG. 10A, acoustic image data is combined with visiblelight image data to generate a display image, which may be presented toa user, for example, via a display. A user may view the display image ofFIG. 10A in order to view which locations in a visible scene areproducing acoustic signals, and the intensities of such signals. Thus, auser may quickly and easily observe which locations are producing soundsand compare the intensities of the sounds coming from various locationsin the scene.

Similar to as described with respect to frequencies elsewhere herein, insome embodiments, acoustic image data may be presented only if thecorresponding acoustic signals meet a predetermined intensity condition.FIG. 10B shows an exemplary display image similar to the display imageof FIG. 10A and including visible light image data and acoustic imagerepresenting acoustic signals above a predetermined threshold. As shown,of locations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090 inFIG. 10A that include acoustic image data, only locations 1020, 1050,1070, and 1090 include acoustic image data representing acoustic signalsthat meet a predetermined condition.

In an exemplary scenario, FIG. 10A can include all acoustic image dataabove a noise floor threshold at each of locations 1010-990, while FIG.10B shows the same scene as FIG. 10A, but only showing acoustic imagedata having an intensity greater than 40 dB. This can help a useridentify which sources of sound in an environment (e.g., in the targetscene of FIGS. 10A and 10B) are contributing certain sounds (e.g., theloudest sounds in a scene).

In addition or alternatively to being compared directly to an intensitythreshold (e.g., 40 dB), as described elsewhere herein, in some suchexamples, predetermined requirements for displaying acoustic image datacan include an amount or rate of change in acoustic intensity at alocation in an image. In some such examples, acoustic image data may bepresented only if a rate of change or an amount of change in acousticintensity at a given location satisfies a predetermined condition (e.g.,is greater than a threshold, less than a threshold, within apredetermined range, etc.). In some embodiments, amount or rate ofchange of acoustic intensity can be palettized and displayed as or inconjunction with intensity acoustic image data. For instance, in anexemplary embodiment, when a rate of change is used as a threshold todetermine in which locations to include acoustic image data, theacoustic image data can include a palettized intensity rate of changemetric for display.

In some examples, a user may manually set an intensity requirement(e.g., minimum value, maximum value, range, rate of change, amount ofchange, etc.) for the acoustic image data to be displayed. As discussedelsewhere herein, including acoustic image data that only meets theintensity requirement can be achieved during acoustic image datageneration (e.g., via filtering received acoustic signals) and/or can beperformed by not displaying generated acoustic image data representingacoustic signals that do not meet the set requirement(s). In some suchexamples, filtering a display image according to intensity values can beperformed after the acoustic image data and visible light image datahave been captured and stored in memory. That is, data stored in memorycan be used to generate display images including any number of filteringparameters, such as only showing acoustic image data meeting predefinedintensity conditions and the like.

In some examples, setting a lower bound for intensity in an acousticimage (e.g., only displaying acoustic image data representative ofacoustic signals above a predetermined intensity) can eliminate theinclusion of undesired background or ambient sounds and/or soundreflections from the acoustic image data. In other instances, setting anupper bound for intensity in an acoustic image (e.g., only displayingacoustic image data representative of acoustic signals below apredetermined intensity) can eliminate the inclusion of expected loudsounds in acoustic image data in order to observe acoustic signalsordinarily masked by such loud sounds.

Several display functions are possible. For example, similar to thefrequency analysis/display discussed with respect to FIG. 9C, in someexamples, the acoustic analysis system can cycle through a plurality ofdisplay images, each showing acoustic image data satisfying differentintensity requirements. Similarly, in some examples, a user may scrollthrough a series of acoustic intensity ranges in order to view thelocations in the acoustic image data having acoustic intensity in thegiven range.

Another parameter that can be used to analyze acoustic data is aperiodicity value of an acoustic signal. FIGS. 11A and 11B showexemplary plots of frequency vs. time of acoustic data in an acousticscene. As shown in the plot of FIG. 11A, the acoustic data includessignals at a frequency X having a first periodicity, signals at afrequency Y having a second periodicity, and signals at a frequency Zhaving a third periodicity. In the illustrated example, acoustic signalshaving different frequencies may also include different periodicity inthe acoustic signals.

In some such examples, acoustic signals can be filtered based onperiodicity in addition or alternatively to frequency content. Forinstance, in some examples, multiple sources of acoustic signals in anacoustic scene may produce acoustic signals at a particular frequency.If a user wishes to isolate one such sound source for acoustic imaging,the user may choose to include or exclude acoustic image data from afinal display image based on the periodicity associated with theacoustic data.

FIG. 11B shows a plot of frequency vs. time of an acoustic signal. Asshown, the frequency increases over time approximately linearly.However, as shown, the signal includes an approximately constantperiodicity over time. Thus, such a signal may or may not appear in anacoustic image depending on selected display parameters. For instance,the signal may at some points in time satisfy a frequency criteria forbeing displayed, but at other points in time, be outside of a displayedfrequency range. However, a user could choose to include or exclude sucha signal from acoustic image data based on the periodicity of thesignals regardless of the frequency content.

In some examples, extracting acoustic signals of a particularperiodicity can be helpful in analyzing a particular portion of a targetscene (e.g., a particular piece of equipment or type of equipment thattypically operates at a certain periodicity). For example, if an objectof interest operates at a certain periodicity (e.g., once per second),excluding signals having periodicity distinct from this can improveacoustic analysis of the object of interest. For example, with referenceto FIG. 11B, if an object of interest operates at periodicity 4,isolating signals having periodicity 4 for analysis may yield improvedanalytics of the object of interest. For example, the object of interestmay emit sounds having periodicity 4, but increasing frequency, such asshown in FIG. 11B. This can imply that the properties of the object maybe changing (e.g., increased torque or load, etc.) and should beinspected.

In an exemplary acoustic imaging process, background noises (e.g., floornoise in an industrial setting, wind in an outdoor environment, etc.)are generally not periodic while certain objects of interest within ascene emit period acoustic signals (e.g., machinery operating at aregular interval). Thus, a user may choose to exclude non-periodicacoustic signals from an acoustic image in order to remove backgroundsignals and more clearly present acoustic data of interest. In otherexamples, a user may be looking to find the source of a constant tone,and so may choose to exclude period signals from acoustic image datathat may obscure viewing of a constant tone. In general, a user maychoose to include in acoustic image data acoustic signals that are abovea certain periodicity, below a certain periodicity, or within a desiredrange of periodicities. In various examples, periodicity can beidentified by either a length of time between periodic signals or afrequency of occurrence of periodic signals. Similar to frequency asshown in FIG. 11B, an analysis of intensity at a given periodicity(e.g., due to an object of interest operating at that periodicity) cansimilar be used to track how acoustic signals from the object changeover time. In general, in some embodiments, periodicity can be used toperform rate-of-change analysis for a variety of parameters, such asfrequency, intensity, and the like.

As describe elsewhere herein, in some examples, various portions of atarget scene can be associated with different distances from an acousticimaging sensor. For example, in some embodiments, distance informationcan include three-dimensional depth information regarding variousportions in a scene. Additionally or alternatively, a user may be ableto measure (e.g., with a laser distance tool) or manually input distancevalues associated with a plurality of locations in a scene. In someexamples, such different distance values for various portions of thescene can be used to adjust the back-propagation calculations at suchlocations to accommodate the specific distance value at that location.

Additionally or alternatively, if different portions of the scene areassociated with different distance values, then proximity from theacoustic sensor array (e.g., measured proximity and/or apparentproximity) can be another differentiable parameter between suchportions. For example, with respect to FIG. 10B, locations 1020, 1050,1070, and 1090 are each associated with a different distance value. Insome examples, similar to frequency or periodicity discussed elsewhereherein, a user can select a particular distance range from which toinclude acoustic image data on a display. For example, a user may selectto only display acoustic image data representative of acoustic signalscloser than a predetermined distance, further than a predetermineddistance, or within a predetermined range of distances.

In addition, in some embodiments, similar to as described with respectto frequencies in FIG. 9C, an acoustic analysis system can be configuredto cycle through a plurality of distance ranges, only showing acousticimage data representing acoustic signals emitted from a location in thetarget scene meeting a current distance range. Such cycling throughvarious displays can help a user visually distinguish informationbetween different acoustic signals. For example, in some cases, objectsmay appear to be close together from the line of sight from anassociated electromagnetic imaging tool (e.g., a visible light cameramodule), and thus acoustic image data combined with electromagneticimage data of such objects may be difficult to distinguish. However, ifthe objects are separated by a depth difference, cycling throughdifferent depth ranges of acoustic image data can be used to isolateeach source of acoustic data from the other.

In general, an acoustic analysis system can be configured to applyvarious settings in order to include and/or exclude acoustic image datarepresentative of acoustic signals that meet one or more predefinedparameters. In some examples, acoustic analysis system can be used toselect a plurality of conditions which must be met by acoustic signalsin order for acoustic image data representative of such signals isdisplayed, for example, in a display image.

For instance, with respect to FIGS. 10A and 10B, only acoustic signalsabove a threshold intensity in the scene of FIG. 10A are shown in FIG.10B. However, additional or alternative restriction is possible. Forexample, in some embodiments, a user may additionally filter theacoustic image data so that acoustic image data is only shown foracoustic signals having frequency content within a predeterminedfrequency range and/or having a predetermined periodicity. In anexemplary embodiment, restricting to predetermined frequencies and/orperiodicities of interest, acoustic image data may be eliminated fromadditional locations, such as 1020 and 1090.

In general, a user can apply any number of acoustic data requirementsfor including or excluding acoustic image data from a display image,including parameters such as intensity, frequency, periodicity, apparentproximity, measured proximity, sound pressure, particle velocity,particle displacement, sound power, sound energy, sound energy density,sound exposure, pitch, amplitude, brilliance, harmonics, rates of changeof any such parameters, or the like. Additionally, in some embodiments,user may combine requirements using any appropriate logicalcombinations, such as AND, OR, XOR, etc. For instance, a user may wishto display only acoustic signals having (intensity above a predeterminedthreshold) AND (frequency within a predetermined range).

Additionally or alternatively, the acoustic analysis system can beconfigured to cycle through one or more parameter ranges to illustratedifferent portions of the target scene, such as shown with respect tocycling through a plurality of frequencies in FIG. 9C. In general, oneor more parameters can be cycled through in such a manner. For instance,a parameter (e.g., intensity) can be separated into a plurality ofranges (e.g., 10 dB-20 dB and 20 dB-30 dB), and the acoustic analysissystem can cycle through such ranges, displaying all acoustic image datafalling within a first range, then all acoustic image data fallingwithin a second range, and so on.

Similarly, in some embodiments, an acoustic analysis system can beconfigured to combine parameter requirements by cycling through nestedranges. For instance, in an exemplary embodiment, acoustic image datathat satisfies a first intensity range AND a first frequency range canbe displayed. The displayed frequency range can be cycled through whilelimiting the displayed acoustic image data to acoustic signalssatisfying the first intensity range. After cycling through thefrequency ranges, the intensity range can be updated to a secondintensity range, such that the displayed acoustic image data satisfiesthe second intensity range and the first frequency range. Similar to theprocess incorporating the first intensity range, the frequency rangescan be similarly cycled through while maintaining the second intensityrange. This process can be continued until all combinations of frequencyranges and intensity ranges have been satisfied. Similar such processescan be performed for any of a plurality of parameters.

Additionally or alternatively, in some embodiments, an acoustic analysissystem can be configured to identify and distinguish a plurality ofsounds in the acoustic scene. For instance, with respect to FIG. 9B, theacoustic analysis system can be configured to identify four discretesounds at locations 910, 920, 930, and 940. The system can be configuredto cycle through a plurality of displays, each showing acoustic imagedata at a single discrete location, similar to as shown in FIG. 9C,though not necessarily dependent on any parameter values. Similarly,such cycling between acoustic image data at individual locations can beperformed after one or more parameter requirements limit the acousticimage data that is displayed.

For example, with respect to FIGS. 10A and 10B, before an intensitythreshold is applied, an acoustic analysis system may cycle through aplurality of acoustic image scenes (e.g., as display images includingthe acoustic image scenes with visible light image data), wherein eachscene includes acoustic image data at a single location. In someembodiments, according to the illustrated example of FIG. 10A, a cycleof 10 separate images, each image including image data at a differentone of locations 1010, 1020, 1030, 1040, 1050, 1060. 1070, 1080, and1090. However, according to some embodiments, after the intensity filteris applied so that only locations having intensity greater than athreshold are displayed (e.g., as in FIG. 10B), the acoustic analysissystem may update the cycling process to only cycle through imagescorresponding to locations that meet the filtering threshold. That is,with respect to FIG. 10B, the cycling process may update to only cyclebetween four images, each showing discrete acoustic image data atlocations 1020, 1050, 1070, and 1090, respectively.

Thus, in various embodiments, each of the locations in a target scenethat includes acoustic image data, either before or after applying oneor more filters to restrict which acoustic image data is shown, is shownin one of a plurality of cycled-through display images. Such cyclicaldisplay of individual acoustic source locations can assist a userviewing the images in identifying the source of particular sound. Insome embodiments, each image in the cycle includes only a single sourceof acoustic data, and in some such embodiments, further includes one ormore parameters of the acoustic data, such as frequency content,intensity, periodicity, apparent proximity, or the like.

In addition or alternatively to cycling between images showing acousticimage data satisfying certain conditions, in some examples, locations ofacoustic signal sources can be detected in acoustic image data anddisplayed in acoustic image data in isolation from other acousticsignals. For example, with respect to FIG. 10A, in some embodiments,acoustic image data representative of acoustic signals emanating fromeach of locations 1010-990 can be identified and cycled through. Forinstance, in an exemplary operating process, display images includingacoustic image data at one of locations 1010-990 can be cycled through,either automatically or at the direction of a user, for individualanalysis of each source of acoustic signals. In various embodiments, theorder in which different locations of acoustic image data are displayedwhile cycling can be dependent on a variety of parameters, such as bylocation, proximity, intensity, frequency content, or the like.

Additionally or alternatively, in some examples, acoustic image datafrom individual locations can by cycled through after applying one ormore filters to isolate only acoustic image data meeting one or morepredetermined conditions. For example, with respect to FIG. 10B,locations 1020, 1050, 1070, and 1090 are shown as including acousticimage data representing acoustic signals meeting a predeterminedintensity requirement. In some embodiments, such a display requirementcan be applied to the individual cycling through of source locations ofacoustic signals. For example, with further reference to FIG. 10B,display images including image data from only one of locations 1020,1050, 1070, and 1090 satisfying an acoustic intensity condition can bycycled through for individual analysis at each location.

In an exemplary process with reference to FIGS. 10A and 10B, acousticimage data collected from a scene can be generally shown in FIG. 10A atlocations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. Suchlocations can include acoustic image data representative of acousticsignals having a variety of acoustic parameters, such as a variety ofintensities, frequency content, periodicity, and the like.

As described elsewhere herein, a user may wish to isolate acousticsignals having one or more particular acoustic parameters, such asacoustic signals having a minimum acoustic intensity. Acoustic imagedata representing acoustic signals not meeting such conditions can beexcluded from the image, for example, leaving acoustic image data atlocations 1020, 1050, 1070, and 1090 as shown in FIG. 10B. However, auser may wish to further identify the source of a particular soundmeeting the display condition (e.g., having intensity above athreshold). Thus, the user may choose to display the acoustic image dataassociated with locations 1020, 1050, 1070, and 1090 one-by-one in orderto view the source location of and analyze each sound individually. Invarious embodiments, the user may choose to cycle manually through suchlocations, or a processor may automatically update the display image tosequentially display acoustic image data of individual locations. Thismay help a user further eliminate and disregard acoustic signals not ofinterest, but that happen to meet one or more filtering parametersapplied to the image.

While described with respect to intensity and FIGS. 10A and 10B, ingeneral, display images including acoustic image data from a singlelocation selected from a plurality of locations can be cycled throughone-by-one for individual analysis. The plurality of locations for whichrepresentative acoustic image data is included can be the entire set oflocations corresponding to sources of acoustic signals in an acousticscene, or can be a subset of such locations, for example, including onlylocations having acoustic signals satisfying one or more conditions.Such conditions can depend on any one or more acoustic parameters, suchas intensity, frequency content, periodicity, proximity, or the like,and can be met based on various parameters being below a predeterminedvalue, above a predetermined value, or within a predetermined range ofvalues.

In various examples, modifying the display image to selectively includeacoustic image data in a display image can be done in a variety of ways.In some embodiments, display images (e.g., including electromagneticimage data and acoustic image data) can be real-time images, in whichelectromagnetic image data and acoustic image data is continuallyupdated to reflect changes in the scene. In some examples, when certainconditions are used to determine whether or not acoustic image data isincluded in the display image, received acoustic signals are analyzed todetermine whether or not to include acoustic image data at variouslocations in the updated real-time image. That is, as new display imageis generated based on newly received acoustic signals andelectromagnetic radiation, the construction of a display image candepend on analysis of the acoustic signals to determine which acousticsignals meet any specific conditions placed on the display image (e.g.,intensity thresholds, etc.). The display image can then be generatedincluding acoustic image data only where appropriate according to suchconditions.

In other examples, display images can be generated from data stored inmemory, such as previously captured acoustic data and electromagneticimage data. In some such examples, the previously-acquired acoustic datais analyzed with respect to various conditions to be placed on theacoustic image data, and is combined with electromagnetic image data inlocations in which the previously-captured acoustic data meets suchconditions. In such embodiments, a single scene can be viewed in manyways, for example, by analyzing different acoustic parameters. Thedisplay image representative of the previously-captured acoustic imagedata can be updated based on any updated conditions placed on thedisplay image for whether or not to include acoustic image data invarious locations in the display image.

In some embodiments, one or more acoustic parameters used to selectivelyinclude acoustic image data in a display image may be used to modify thedisplay image and/or image capturing techniques. For example, inreal-time imaging examples, various conditions for determining whetheror not to include acoustic image data in a display can includedistance-to-target (e.g., apparent distance or measured distance) and/orfrequency content. As described elsewhere herein, some such parameterscan be used in selecting an acoustic sensor array and/or processingscheme for generating acoustic image data. Thus, in some such examples,when acoustic image data is only represented based on such parametersmeeting one or more predetermined conditions, an acoustic sensor arrayand/or a processing scheme for generating acoustic image data can beselected based on such conditions.

For example, in an exemplary embodiment, if acoustic image data is onlyto be included in a real-time image in locations at which correspondingacoustic signals include frequency content within a first frequencyrange, one or more acoustic sensor arrays can be selected for acquiringacoustic signals that are best suited for the first frequency range.Similarly, if acoustic image data is only to be included in a real-timeimage at locations in which a source of acoustic signals is within afirst distance range, one or more acoustic sensor arrays can be selectedfor acquiring acoustic signals that are best suited for acoustic imagingin the first distance range. Additionally or alternatively, asdescribed, for example, with respect to FIG. 6 , processing schemes forgenerating acoustic image data can be selected based on desiredfrequency or distance conditions. Such selected acoustic imaging sensorarray(s) sand processing schemes can be subsequently used to receiveacoustic signals and generate acoustic image data for the updatedreal-time display image in order to optimize the acoustic image datathat is included.

Similarly, in some embodiments in which a display image is generatedfrom historical data previously stored in memory, various conditionsdetermining in which locations to include acoustic image data in thedisplay image can be used to update the acoustic image datarepresentative of the acoustic scene. For instance, in some embodiments,data stored in memory comprises raw acoustic data received by theacoustic sensor array(s) from the time the acoustic signals werereceived. Based on the conditions for determining whether or notacoustic image data is included at various locations in the displayimage (e.g., desired distance and/or frequency ranges), a processingscheme (e.g., a back-propagation algorithm) can be selected for use withthe raw data stored in memory for generating acoustic image dataoptimized to the desired parameters to be displayed.

It will be appreciated that, while generally described and shown usingvisible light image data and acoustic image data, the processesdescribed with respect to FIGS. 9A-C, 10A and 10B can be used includingany of a variety of electromagnetic image data. For example, in variousembodiments, similar processes could be performed with infrared imagedata or ultraviolet image data instead of visible light image data.Additionally or alternatively, combinations of electromagnetic spectrumscan be used in such processes, such as blended infrared image data andvisible light image data. In general, in various examples, acousticimage data can be selectively shown (e.g., included when correspondingacoustic signals meet one or more predetermined parameters) incombination any combination of electromagnetic image data.

In some embodiments, an acoustic analysis system is configured to storeone or more acoustic signals and/or acoustic image data in a database,for example, in local memory and/or accessible from an external orremote device. Such acoustic signals can include acoustic image datarepresentative of an acoustic scene during normal operation and/or otherparameters associated with an acoustic scene, such as frequency data,intensity data, periodicity data, and the like. In various examples,database scenes can include acoustic image data and/or other acousticparameters (e.g., intensity, frequency, periodicity, etc.)representative of a broad scene (e.g., a factory) and/or a more specificscene (e.g., a particular object).

In some embodiments, a database scene can be generic to a particulartype of equipment, such as a particular model of equipment. Additionallyor alternatively, database scenes can be specific to individual objects,even if different such objects are different instances of the sameobject (e.g., two separate machines that are the same model). Similarly,database scenes can be more specific, for example, including aparticular operating state of an object. For instance, if a particularobject has multiple modes of operation, a database can include multiplescenes of such an object, one for each of the modes of operation.

In various embodiments, database scenes can be a single acoustic imageand/or associated acoustic parameters. In other examples, databasescenes can include composite data formed from a plurality of previouslycaptured acoustic images and/or associated parameters. In general,database scenes (e.g., acoustic images and/or parameters) can include anacoustic representation of the scene during normal operation. In someexamples, the database can include other elements associated with thescene, such as a corresponding visible light image, infrared image,ultraviolet image, or combinations thereof, for example. In someembodiments, database generation and/or comparisons can be performedsimilar to the database generation and comparisons of infrared imagedata described in U.S. patent application Ser. No. 15/190,792, filedJun. 23, 2016, and entitled “THERMAL ANOMALY DETECTION,” which isassigned to the assignee of the instant application and is herebyincorporated by reference in its entirety. In some embodiments, adatabase can be generated by capturing acoustic image data and/or one ormore associated acoustic parameters (e.g., frequency, intensity,periodicity, etc.) of a scene while objects within the scene areoperating correctly. In some such examples, a user may tag the captureddatabase image to associate the image with one or more objects,locations, scenes, or the like, so that the captured acoustic imageand/or associated parameter(s) can be identified in the future fordatabase analysis and comparisons.

Newly generated acoustic image data can be compared to acoustic imagedata stored in the database to determine whether or not the acousticprofile of the acoustic scene is within typical operating standards.Additionally or alternatively, acoustic parameters, such as intensity,frequency, periodicity, and the like, from a live acoustic scene and/ora newly-generated acoustic image can be compared to similar parametersin the database.

Comparing current acoustic image data to historical acoustic image data(e.g., a previously-captured image, a composite image generated from aplurality of previously-captured images, a factory-provided expectedimage, etc.) stored in a database can be done a plurality of ways. FIGS.12A-12C show multiple exemplary ways for comparing acoustic image datato historical acoustic image data stored in a database. FIG. 12A showsan acoustic imaging tool 1200 including an acoustic sensor array 1202having an acoustic field of view 1212 and an electromagnetic imagingtool 1204 having an electromagnetic field of view 1214. As shown, theelectromagnetic field of view 1214 and the acoustic field of view 1212include a target scene 1220 including an object of interest 1222. Insome embodiments, the acoustic imaging tool 1200 is permanently fixed ina location such that the object of interest 1222 is in theelectromagnetic field of view 1214 and the acoustic field of view 1212.In some embodiments, the acoustic imaging tool 1200 can be powered viainductive or parasitic power, can be wired into AC main power in abuilding, or the to be configured to continually monitor object 1222.

Fixed acoustic imaging tool 1200 can be configured to periodicallycapture acoustic and/or electromagnetic image data of object 1222.Because the acoustic imaging tool 1200 is generally fixed in place,images captured at different times will be from approximately the samevantage point. In some examples, acoustic image data captured viaacoustic imaging tool 1200 can be compared to a database of acousticimage data representative of approximately the same scene, for example,to detect anomalies or abnormalities in the acoustic scene. This can beperformed, for example, as described in U.S. patent application Ser. No.15/190,792, which is incorporated by reference.

FIG. 12B shows an exemplary display, for example, on a handheld acousticimaging tool. The display 1230 includes two sections, 1232 and 1234. Inthe illustrated example, section 1234 shows a database image 1244 of anobject of interest, while section 1232 comprises a live display 1242 ofreal-time acoustic image data of an object. In such a side-by-side view,a user may compare the live image 1242 to the database image 1244 inorder to view any differences between a typical acoustic signals (e.g.,as shown in database image 1244) and the current real-time image 1242.Similarly, the user can compare if the live image 1242 approximatelymatches the database image 1244. If so, the user may capture the liveacoustic image for further analysis and/or comparison to the databaseimage 1244.

FIG. 12C shows another exemplary display, for example, on a handheldacoustic imaging tool. The display 1250 of FIG. 12C shows a databaseimage 1254 and a live image 1256 on the same display 1252. In theexample of FIG. 12C, a user can similarly compare the acoustic imagedata in the live image 1256 to the acoustic image data in the databaseimage 1254 in order to view differences. Additionally, the user mayadjust alignment of the acoustic imaging tool in order to align theobject in the live image 1256 with the object in the database image 1254for further analysis and comparison.

As a result of the processes in FIGS. 12A-12C, live and/or recentlycaptured acoustic images can be compared to previous acoustic imagedata, such as from a database. In some examples, such processes can beused to register the live and/or recently captured acoustic image withthe database image for automated comparison. Other processes that can beused to “recapture” acoustic image data from a similar vantage point asthe database image are described in U.S. patent application Ser. No.13/331,633, filed Dec. 20, 2011, and entitled, “THERMAL IMAGING CAMERAFOR INFRARED REPHOTOGRAPHY,” U.S. patent application Ser. No.13/331,644, filed Dec. 20, 2011, and entitled, “THERMAL IMAGING CAMERAFOR INFRARED REPHOTOGRAPHY,” and U.S. patent application Ser. No.13/336,607, filed Dec. 23, 2011, and entitled, “THERMAL IMAGING CAMERAFOR INFRARED REPHOTOGRAPHY,” each of which is assigned to the assigneeof the instant application and is incorporated by reference in itsentirety.

Comparing real-time acoustic image data and/or acoustic signatures to acorresponding acoustic image and/or acoustic signature of a comparablescene/object can be used to provide fast and simplified analysis of thestate of operation of the scene/object. For example, a comparison mayindicate that certain locations within the acoustic scene are emittingacoustic signals that have a different intensity or frequency spectrumthan during typical operation, which can be indicative of a problem.Similarly, locations in the scene may be emitting acoustic signals thatare typically silent. Additionally or alternatively, comparison ofoverall acoustic signatures of a live scene and a historic scene from adatabase can generally indicate changes in acoustic parameters in thescene, such as frequency content, acoustic intensity, and the like.

In some examples, an acoustic analysis system is configured to comparethe recent/real-time acoustic scene with a database. In someembodiments, the acoustic analysis system is configured to characterizethe differences between the recent/real-time scene and the databasescene and diagnose one or more possible problems in the current scenebased on the comparison. For instance, in some embodiments, a user maypre-select an object of interest or a target scene for comparison to anacoustic database. The acoustic analysis system can, based on theselected object/scene, compare the database image and/or otherparameters to the recent/current image and/or other parameters toanalyze the scene. Based on the selected object/scene from the database,the acoustic analysis system may be capable of identifying one or moredifferences between the database image/parameters and the recent/currentimage/parameters and associate the identified difference(s) with one ormore likely causes of the differences.

In some examples, the acoustic analysis system can be pre-programmedwith a plurality of diagnostic information, for example, associatingvarious differences between database images/parameters andrecent/current images/parameters with likely causes and/or solutions tocauses. Additionally or alternatively, a user may load such diagnosticinformation, for example, from a repository of diagnostic data. Suchdata may be provided, for example, by a manufacturer of the acousticanalysis system, the manufacturer of an object of interest, or the like.In still further examples, an acoustic analysis system can additionallyor alternatively learn diagnostic information, for example, via one ormore machine learning processes. In some such examples, a user maydiagnose one or more issues in a target scene after observing acousticdeviations of the scene from typical, and may input data representativeof the one or more issues and/or one or more solutions into the acousticanalysis system. The system can be configured to, over time and viamultiple data entries, learn to associate different discrepanciesbetween recent/current images and/or parameters and those stored in adatabase with certain problems and/or solutions. Upon diagnosing anissue and/or determining a proposed solution, the acoustic analysissystem can be configured to output a suspected problem and/or proposedsolution to a user, for example, via a display. Such a display can be ona handheld acoustic inspection tool or a remote device (e.g., a user'ssmartphone, tablet, computer, etc.). Additionally or alternatively, sucha display indicating a potential problem and/or solution can becommunicated to a remote site, such as an off-site operator/systemmonitor, for example, via a network.

In some example diagnostic characterizations, an acoustic analysissystem may observe a particular periodic squeaking sound indicatingadditional lubrication is needed on an operating machine. Similarly, aconstant, high-pitched signal could indicate a gas or air leak in atarget scene. Other issues may similarly have recognizable acousticsignatures, such as a broken bearing within an object under analysis,such that viewing the acoustic signature via an acoustic imaging system(e.g., a handheld acoustic imaging tool) can help diagnose anyabnormalities in a system or object.

An acoustic analysis system capable of comparing received acousticsignals to a baseline (e.g., acoustic image data and/or parameters froma database) and performing diagnostic information and/or suggesting acorrective action can eliminate the need for an experienced expert toanalyze acoustic data of a scene. Rather, an acoustic inspection andanalysis can be performed by a system operator with limited or noexperience in analyzing acoustic data.

FIG. 13 is a process-flow diagram showing exemplary operation ofcomparing received acoustic image data to a database for objectdiagnostics. The method includes receiving a selection of a target ofinterest (1380) and retrieving a baseline acoustic image and/or acousticparameters of the target of interest from a database (1382). Forexample, a user may wish to perform acoustic analysis of a particularobject of interest, and may select such an object from a predefined listof objects having available baseline acoustic image and/or parametersavailable in the database.

The method further includes the step of capturing acoustic image dataand associated parameters representative of the target of interest(1384), for example, using a handheld acoustic imaging tool. Aftercapturing the acoustic image data and associated parameters (1384), themethod includes comparing the captured acoustic image data and/orassociated parameters to retrieved baseline image and/or parameters(1386).

The method of FIG. 13 further includes, if the captured acoustic imagedata and/or parameters deviate sufficiently from the baseline (1388),diagnosing operation issues of the target of interest based on thecomparison (1390). The method can further include the step of displayingan indication of possible issues and/or corrective actions to a user(1392). In some embodiments, the a comparison display, for example, adifference image showing the difference between the current acousticimage data and baseline acoustic image data can be additionally oralternatively displayed to a user.

In some such examples determining if there is deviation from thebaseline (1388) comprises comparing one or more acoustic parameters ofthe captured data to like parameters in the baseline data anddetermining if the difference between the captured and baselineparameters exceeds a predetermined threshold. In various examples,different parameters may include different thresholds, and suchthresholds can be absolute thresholds, statistical thresholds, or thelike. In some embodiments, comparisons can be done on alocation-by-location basis, and may be performed for a subset oflocations within a scene.

For example, with respect to FIG. 9B, it is possible that only locationsincluding acoustic image data and appearing on the object (e.g.,locations 910 and 940) are analyzed with respect to operation of theobject. In such an example, different acoustic parameters at each oflocations to be compared (e.g., 910 and 940) are compared individuallybetween captured and database images. For example, comparing thecaptured data and/or associated parameters to those from the databasecan include, with reference to FIG. 9B, comparing the frequency,intensity, and periodicity of location 910 in the captured image to thefrequency, intensity, and periodicity, respectively, of location 910 inthe database image. Similar comparisons can be performed at location 940between the captured image and the database image. As described, eachcomparison can include a different metric for determining if there issufficient deviation from the baseline (1388).

Diagnosing operation issues (1390) and displaying an indication ofpossible issues and/or corrective actions (1392) can be performed basedon the combination of comparisons between captured and baseline imagedata and/or parameters. In some examples, such diagnostics can include amulti-dimensional analysis, such as combining comparisons of multipleparameters at a given location. For instance, in an exemplaryembodiment, a certain condition might be indicated by both a deviationin frequency from the baseline that is greater than a first thresholdand a deviation in intensity from the baseline that is greater than asecond threshold.

In some examples, even after displaying an indication of possible issuesand/or corrective actions (1392), the process can include capturing newacoustic image data and associated parameters (1384) and repeating thecomparison and diagnostic processes. Thus, a user may observe whether ornot any taken corrective actions are effectively changing the acousticsignature of the object in order to rectify an identified issue and/orbring the acoustic signature of the object into conformity with thebaseline.

In some embodiments, if, after comparing the captured data to thebaseline data (1386), there is not a sufficient deviation from thebaseline (1388), the process may end (1394) with the conclusion that,based on the current acoustic signature of the object, the object isoperating normally. Additionally or alternatively, new acoustic imagedata and associated parameters of the target of interest can be captured(1384) and the comparison and diagnostic process can be repeated. Insome examples, continued repeated analysis can be performed using afixed acoustic analysis system, for example, including the acousticimaging tool 1200 in FIG. 12A.

Comparisons of acoustic data (e.g., image data and/or other acousticparameters) can help a user more easily identify if an object isfunction correctly, and if not, to diagnose issues with the object. Insome examples, comparing to a baseline can help a user disregard“normal” sounds in a scene, such as expected operating sounds orfloor/background sounds that may be irrelevant to an operating issue ofthe object.

During operation, observation of acoustic image data and/or associatedacoustic parameters or observing the results of a comparison betweencurrent and database acoustic scenes may indicate locations of interestto a user for further inspection. For example, a comparison acousticimage showing deviations from the database image may indicate one ormore locations in scene that are operating abnormally. Similarly,viewing an acoustic image having an acoustic signature at one or morelocations that are unexpected may indicate a location of interest to auser. For example, with reference to FIG. 10B, a user observing FIG. 10Bon a display of an acoustic imaging system may realize that a particularlocation (e.g., 1020) is unexpected emitting acoustic signals, orsimilarly, a comparison to a baseline image indicates unexpectedparameters of the acoustic signals at that location (e.g., unexpectedfrequency, intensity, or the like).

In some such examples, the user may move closer to such a location inorder to more closely inspect the location for abnormalities. Uponmoving closer to the object, the distance-to-target value may be updatedto reflect the new distance between an acoustic array and the targetlocation. The acoustic sensor array and/or the back-propagationalgorithm may be updated based on the updated distance-to-target.Additionally or alternatively, updated acoustic analysis from a closerlocation may yield different analysis of the acoustic signals from thetarget. For instance, high frequency acoustic signals (e.g., ultrasoundsignals) tend to attenuate over a relatively short distance from thesource of the acoustic signals. Thus, as a user moves closer to thetarget for further inspection, additional signals (e.g., high frequencysignals) may be visible to the acoustic sensor array. Such apparentchanges in the observable scene may also result in adjusting theacoustic sensor array and/or the back-propagation algorithm used foracoustic imaging.

Accordingly, the sensor array and/or back-propagation algorithm used foracoustic imaging can be updated one or more times as the user movescloser to an object or region of interest. Each update may provideadditional details regarding the object or region of interest that maynot have been observable from a further distance away using a differentsensor array and/or back-propagation algorithm. Moving closer to anobject or region of interest, for example, based on initial observationsof a broader scene, can also increase the acoustic intensity of theacoustic signals of interest relative to background sounds in theenvironment.

In some embodiments, an acoustic analysis system (e.g., a handheldacoustic imaging tool) can prompt a user to move more closely to anobject or regions of interest within a scene. For example, uponcomparing a current acoustic image to a baseline database image, theacoustic analysis system may identify one or more locations in the scenethat deviate from baseline. The acoustic analysis system may highlightsuch one or more locations to a user, for example, via a display, andsuggest the user move closer to the identified location(s) for furtheranalysis. In some examples, the acoustic analysis system can classifythe identified location, such as a sub-component of an object or aparticular object within an environment, as having its own baselineprofile stored in a database. The system may be configured to suggestand/or implement such a profile of the classified location to facilitatefurther analysis of the identified location when the user moves closerfor additional inspection.

Systems and processes described herein can be used to improve the speed,efficiency, accuracy, and thoroughness of acoustic inspections. Variousautomated actions and/or suggestions (e.g., of a sensor array, aback-propagation algorithm, etc.) can increase the ease of inspection tothe point that an inexperienced user may perform a thorough acousticinspection of an acoustic scene. Moreover, such processes can be used toanalyze a broad scope of scenes, such as entire systems, individualobjects, and sub-components of individual objects. Predefined and/oruser-generated profiles of baseline acoustic data of acoustic scenes canassist even inexperienced users in identifying abnormalities in capturedacoustic data.

Registration of acoustic image data with other data streams, such asvisible light, infrared, and/or ultraviolet image data, can provideadditional context and detail to what objects are emitting acousticsignals represented in acoustic image data. Combining acoustic sensorarrays and a distance measuring tool (e.g., a laser distance finder) canassist a user in quickly and easily determining a properdistance-to-target value for use during acoustic imaging processes. Invarious examples, an acoustic sensor array, distance measuring tool,processor, memory, and one or more additional imaging tools (e.g.,visible light camera module, infrared camera module, etc.) can besupported by a single housing in a handheld acoustic imaging tool thatcan provide efficient acoustic analysis of a plurality of scenes. Such ahandheld acoustic imaging tool can be moved from scene to scene forrapid analysis of multiple objects of interest. Similarly, using ahandheld tool, a user can move closer to a location of interest within ascene for further inspection or analysis.

Various systems and methods for performing acoustic imaging andgenerating and displaying acoustic image data are described herein.Exemplary systems can include an acoustic sensor array that includes aplurality of acoustic sensor elements configured to receive acousticsignals from an acoustic scene and output acoustic data based on thereceived acoustic signals.

Systems can include an electromagnetic imaging tool configured toreceive electromagnetic radiation from a target scene and outputelectromagnetic image data representative of the receivedelectromagnetic radiation. Such imaging tools can include an infraredimaging tool, a visible light imaging tool, an ultraviolet imaging tool,or the like, or combinations thereof.

Systems can include a processor in communication with the acousticsensor array and the electromagnetic imaging tool. The processor can beconfigured to receive electromagnetic image data from theelectromagnetic imaging tool and acoustic data from the acoustic sensorarray. The processor can be configured to generate acoustic image dataof a scene based on the received acoustic data and received distanceinformation representative of a distance to target, for example, via aback-propagation calculation. The acoustic image data can include avisual representation of the acoustic data such as by a palette or colorscheme such as described elsewhere herein.

The processor can be configured to combine the generated acoustic imagedata and the received electromagnetic image data to generate a displayimage comprising both acoustic image data and electromagnetic imagedata, and communicate the display image to a display. Combining theacoustic image data and the electromagnetic image data can includecorrecting a parallax error between the acoustic image data and theelectromagnetic image data, for example, based on the received distanceinformation.

In some examples, distance information can be received from a distancemeasuring tool in communication with the processor. Distance measuringtools can include, for example, an optical distance measuring device,such as a laser distance measuring device, and/or an acoustic distancemeasurement device. Additionally or alternatively, a user can enterdistance information manually, for example, via a user interface.

Systems can include a laser pointer to help identify locations of pointsof interest such as sounds or sound profiles based upon selectedparameters such as frequency, decibel level, periodicity, distance, orthe like, or combinations thereof. Such a laser pointer can be used topinpoint and align the field of view of the scene with the appropriatesound visualization as displayed on the display. This may be useful inenvironments where an object under inspection is at a distance relativeto the acoustic imaging device or if it is not clear where thevisualization of the sound on the display is relative to the actualscene.

In some examples, the laser pointer can be visualized on the display.Such visualization can include the generating a laser pointer spot(e.g., via the processor) on the display representative the laserpointer in the actual scene. In some examples, the position of the laserpointer can be enhanced on the display, for instance, with an iconrepresentative of the laser pointer in the scene or another aligneddisplay marker to better determine the location on the display relativeto the actual scene.

As described elsewhere herein, a thermal imaging system can beconfigured to create a false-color (e.g., palettized), symbolic, orother non-numerical visual representation of acoustic data generated byone or more acoustic sensors, such as by creating acoustic image data.Additionally or alternatively, a system can provide a user with audiofeedback, such as via speakers, headphones, a wired orremotely-communicating headset, or the like. The transmission of suchaudio or heterodyne audio can be synchronized to the visualrepresentation of the detected and displayed sounds.

In various examples, acoustic data can be visualized in a variety ofways, for example, to facilitate understanding of such data and preventa viewer from making incorrect assumptions about the nature of thesounds being visualized. In some examples, different types ofvisualization can provide an intuitive understanding of the visualizedsounds.

In some embodiments, a generated display includes a non-numerical visualrepresentation with contextual numerical and/or alphanumerical data inorder to provide a thorough presentation of information regarding soundsbeing visualized, which can assist a user in determining and/orimplementing one or more appropriate courses of action.

Various display features, including various non-numeric graphicalrepresentations (e.g., symbols, palettization, etc.) and alphanumericinformation can be combined. In some embodiments, the display featurespresent in a given representation of a scene can be customized by auser, for instance, from a plurality of selectable settings.Additionally or alternatively, preset combinations of display featurescan be selectable by a user to automatically include a desiredcombination of information in a display image. In various embodiments,aspects of a display image are adjustable by a user, for example, via avirtual interface (e.g., provided via a touchscreen) and/or via physicalcontrols.

FIG. 14 shows a visualization of acoustic data using a gradientpalettization scheme. As shown, an acoustic parameter (e.g., intensity)is shown via a gradient palettization scheme. In an exemplary gradientpalettization scheme, a parameter value will have a unique colorassociated therewith according to the palettization scheme. A change inthe parameter value at a given pixel will generally result in a changein the color associated with that pixel to represent the new parametervalue. As shown in the example of FIG. 14 , the acoustic parameter valueappears to change radially from a center position in acoustic signal atpositions 232, 234, 236. Other examples of gradient palettization aredescribed U.S. patent application Ser. No. 15/802,153, filed Nov. 2,2017, and assigned to the assignee of the instant application.

FIG. 15 shows a visualization of acoustic data using a plurality ofshaded concentric circles. In some such examples, as opposed to agradient palettization scheme having a color associated with a parametervalue, each solid color the concentric circles shown in FIG. 15 canrepresent pixels having acoustic parameter values within a range ofvalues associated with that color. In the illustrated examples, anacoustic parameter (e.g., intensity) associated with acoustic signals atlocations 332, 334, 336 changes radially from the center of the acousticsignal. In an exemplary palettization scheme, pixels shown in redrepresent an acoustic parameter value within a first range of parametervalues, pixels shown in yellow represent an acoustic parameter valuewithin a second range of parameter values, and pixels shown in greenrepresent an acoustic parameter value within a third range of parametervalues, however, other display techniques are possible, includingadditional or alternative colors, patters, or the like. In variousembodiments, ranges of values can correspond to absolute ranges, such asintensity values between 10 dB and 20 dB, or can be relative ranges,such as intensity values between 90% and 100% of the maximum intensity.

As described elsewhere herein, in some embodiments, a display imageincluding electromagnetic image data and acoustic image data can includeboth a visual indication of an acoustic signal and an alphanumericrepresentation of one or more parameters associated with the acousticsignal. FIG. 16 shows an exemplary visualization showing bothnon-numeric information (e.g., palettization via parameter value ranges)and alphanumeric information. In the illustrated example, analphanumeric sound intensity value label is associated with each ofthree locations having palettized acoustic image data (e.g., intensitydata). As shown, acoustic signals have corresponding visual indicatorsrepresenting an acoustic parameter associated therewith (1602, 1604,1606), as well as alphanumeric information (1612, 1614, 1616,respectively). In an exemplary embodiment, the alphanumeric informationcan provide a numerical value, such as maximum intensity value,associated with the location at which the palettized acoustic data isdisplayed. In some examples, a user may select one or more locations atwhich to display the palettization and/or the alphanumeric data. Forexample, a user may choose to annotate a display image usingalphanumeric representations of an acoustic parameters associated withone or more acoustic signals within the scene.

In some examples, alphanumeric information can represent a plurality ofparameters (e.g., acoustic parameters) associated with an acousticsignal at a given location in a scene. FIG. 17 shows an examplevisualization including both non-numeric information (e.g.,palettization via parameter value ranges) and alphanumeric information.In the illustrated example, a sound intensity value and a correspondingfrequency value (e.g., an average frequency or a peak frequency) areshown in alphanumeric information 1712, 1714, 1716 associated with eachof three locations having palettized acoustic data (e.g., intensitydata) shown via indicators 1702, 1704, 1706, respectively. Similar to asdiscussed with respect to FIG. 16 , the inclusion of various such dataat various locations can be initiated by a user. For example, a user maychoose to annotate a display image using alphanumeric representations ofone or more acoustic parameters associated with one or more acousticsignals within the scene.

FIG. 18 shows another exemplary visualization showing both non-numericinformation (e.g., palettization via parameter value ranges) andalphanumeric information. In the illustrated example, a distancemeasurement is included with alphanumeric information 1812, 1814, 1816associated with each of three locations having palettized acoustic data(e.g., intensity data), shown via indicators 1802, 1804, 1806,respectively. Similar as discussed with respect to FIG. 16 , theinclusion of various such data at various locations can be selected by auser, for example, as part of a display image annotation.

In some examples, non-numeric representations can be used to communicateinformation related to a plurality of acoustic parameters. For instance,FIG. 19 shows an exemplary visualization showing indicators 1902, 1904,1906 (in this case, circles) of different size and color representativeof different acoustic parameter values. In an exemplary embodiment, thesize of the indicator corresponds to the intensity of the acousticsignal at a given location, while the color of the indicator correspondsto a peak or average frequency. In an exemplary embodiment, indicatorsize can show relative values such that comparing the size of oneindicator to the size of another represents the relative differencebetween the represented acoustic parameter values at the locationsassociated with the indicators. Additionally or alternatively,alphanumeric information can be included to provide absolute or relativeacoustic parameter values.

In some embodiments, a colorized indicator can be used to represent theseverity of one or more detected acoustic signals and/or associatedacoustic parameters, such as an amount of deviation from a baselineparameter. FIG. 20 shows an exemplary visualization showing a pluralityof indicators 2002, 2004, 2006 having different colors indicative of theseverity indicated by acoustic signals from the corresponding locations.For instance, in an exemplary embodiment, a red indicator indicates acritical severity based on one more acoustic parameters (e.g., whencompared to a baseline, such as a baseline of typical operatingconditions), a yellow indicator indicates moderate severity, and a greenindicator represents minor severity. In other examples, other colorschemes or appearance characteristics (e.g., indicator transparency,indicator size, etc.) can be used to visually distinguish severity of anacoustic signal. In the illustrated example, indicator 2004 representsgreatest level of severity, indicator 2006 represents the next greatest,and indicator 2002 represents the least severe acoustic signal.

As described elsewhere herein, in various embodiments, one or moreacoustic parameters can be displayed on a visual representation of anacoustic scene, for example, by way of a palettized color or grayscaledisplay. In some embodiments, a system can be configured to identify oneor more locations in a scene meeting one or more acoustic conditions,such as an identified frequency range, intensity range, distance range,or the like. In some examples, various locations corresponding to anacoustic profile (e.g., satisfying a particular set of conditions orparameters) can be identified. Such identified locations can bepresented in a distinguishing manner from the acoustic image datapalettization scheme otherwise used in creating a display image. Forexample, FIG. 21 shows a scene including indicators at a plurality oflocations within the scene. Indicators 2101, 2102, 2103, 2104, and 2105are positioned within the scene. Indicators 2103, 2104, and 2105 includepalettized acoustic image data, for example, representing values of oneor more acoustic parameters corresponding to scale 2110. Indicators 2101and 2102 are shown having a unique presentation scheme distinguishablefrom the palettization scheme appearing at locations 2103, 2104, and2105. In such an embodiment, a user may quickly and easily identifythose locations in an image satisfying one or more desired conditions.In some such examples, a user may select the one or more desiredconditions for displaying in a distinguishing way based on a selectionof a range of values, for example, from a scale such as 2110.

Additionally or alternatively, a location meeting the conditions of aparticular sound profile can be presented with an icon representative ofthe met condition, such as a corresponding acoustic profile. Forexample, FIG. 22 shows a plurality of icons 2202, 2204, 2206, 2208positioned within a display image indicating recognized acousticprofiles within the scene. Exemplary profiles as shown in FIG. 22include bearing wear, air leak, and electrical arcing. Such profiles canbe identified by acoustic signals satisfying a set of one or moreparameters associated with such profiles in order to be categorized intosuch profiles.

FIG. 23 shows another exemplary display showing acoustic data via aplurality of indicators 2302, 2304, and 2306 using concentric circlesand alphanumeric information representing acoustic intensity associatedwith each of the acoustic signals. As described elsewhere herein, insome examples, the size of an indicator can represent one or moreacoustic parameters present at the corresponding location. In someembodiments, indicators can be monochromatic, and indicate acousticparameters in one or more other ways, such as by indicator size, lineweight, line type (e.g., solid, dashed, etc.).

In some examples, a display can include alphanumeric information basedon a selection made by a user. For instance, in some embodiments, asystem (e.g., via a processor) can include information representing oneor more acoustic parameters of an acoustic signal located at aparticular location in response to a user selection of an indicator on adisplay (e.g., via a user interface) at such a location. FIG. 24 showsan example display image having an indicator and additional alphanumericinformation associated with the represented acoustic signal. In anexample, indicator 2402 on a display can be selected (e.g., representedvia crosshairs, which can be indicative of a selection, such as via atouch screen input) for further analysis. The display shows alphanumericinformation 2404 including a list of data associated with the locationcorresponding to the indicator, including a peak intensity andcorresponding frequency, a frequency range, a distance to locationmeasurement, and a level of criticality indicated by the acousticsignals from that location.

In some examples, a display image can include a plurality of indicatorsrepresenting a corresponding plurality of acoustic signals in the scene.In some embodiments, in such a case, a user may select one or moreindicators (e.g., via a touchscreen or other user interface), and inresponse to detecting the selection, the processor can presentadditional information regarding the acoustic signal. Such additionalinformation can include an alphanumeric of one or more acousticparameters. In some examples, such additional information can bedisplayed for multiple acoustic signals simultaneously. In otherexamples, such additional information for a given acoustic signal ishidden when another acoustic signal is selected.

As described elsewhere herein, in some examples, systems can include alaser pointer. In some examples, laser pointers can have a fixedorientation, or can have an adjustable pointing, for example,controllable via the processor. In some examples, the system can beconfigured to aim the laser pointer at a location in the target sceneassociated with a selected location in an image. FIG. 25A shows a system(in some examples, embodied as a handheld tool) including a display,such as the display shown in FIG. 24 , in which an indicator 2502 isselected. A laser pointer 2504 emits a laser beam 2506 toward the scene,wherein the laser creates a laser spot 2508 in the scene correspondingto the location of the selected indicator 2502 in the image. This canhelp a user visualize the location of the selected and/or analyzedacoustic signals in the environment. In some embodiments, the laser spot2508 is detectable by an electromagnetic imaging tool, and is visible onthe display along with the displayed indicator 2502 and alphanumericinformation 2512 including acoustic parameter information. In someexamples, the acoustic imaging system is configured to detect or predictthe location of the laser in the scene, and provide a visual indication2510 of the laser location. FIG. 25B shows a display image such as thatshown in the system view of FIG. 25A.

In embodiments in which the laser pointer has a fixed orientation, theuser can view the display image having a visual indication of the laserlocation as feedback so that the user can adjust the pointing of thelaser to coincide with the selected acoustic signal.

As described elsewhere herein, in some embodiments, acoustic image datacan be combined with electromagnetic image data for presentation in adisplay image. In some examples, the acoustic image data can include anadjustable transparency such that various aspects of the electromagneticimage data are not completely obscured by FIG. 26 shows acoustic imagedata represented by an indicator 2602 at a location in the scene,wherein the indicator 2602 includes a gradient palettization scheme. Asystem can include a display device, which can be integral with orseparate from an acoustic imaging tool, configured to present displaydata including electromagnetic image data and acoustic image data.

In some embodiments, the device (e.g., a handheld acoustic imaging tool)can include a physical blending control 2614 (e.g., one or more buttons,knobs, sliders, etc., which can be included as part of a user interface)and/or a virtual blending control 2604, such as via a touchscreen orother virtually-implemented interface. In some embodiments, suchfunctionality can be provided by an external display device, such as asmartphone, tablet, computer, or the like.

FIG. 27 shows a virtual and/or physical blending control tools for adisplay image including a partially-transparent concentric circlepalettization scheme. Similar to as described with respect to FIG. 26 ,an indicator 2702 can represent an acoustic signal within a scene. Anacoustic imaging system can include a physical blending control 2714and/or a virtual blending control 2704 that can be used to adjust thetransparency of the acoustic image data (e.g., indicator 2702) withinthe display image.

Additionally or alternatively, physical and/or virtual interfaces can beused to adjust one or more display parameters. For instance, in someembodiments, one or more filters can be applied to selectively displayacoustic image data satisfying one or more conditions, such as describedelsewhere herein. FIG. 28 shows a scene including an indicator 2802having a gradient palettization indicating a location in the scenesatisfying one or more filters (e.g., having one or more acoustic orother parameters meeting one or more corresponding thresholds orpredetermined conditions). In various examples, the filtering can beselected and/or adjusted via a physical control 2814 (e.g., via one ormore buttons, knobs, switches, etc.) and/or a virtual control 2804(e.g., a touchscreen). In the illustrated example, the filter includesdisplaying acoustic image data for only those acoustic signals having anacoustic parameter (e.g., frequency) falling within a predefined range2806 of the acoustic parameter. As shown, the predefined range 2806 is asubset of possible filter ranges 2816. In some examples, a user mayadjust the limits of the predefined range 2806, for example, via virtual2804 or physical 2814 control to adjust the effects of the filter.

FIG. 29 shows a virtual and/or physical filter adjustment for a displayimage including a partially-transparent concentric circle palettizationscheme. As shown, indicator 2902 is shown within the scene based onacoustic parameters falling within a predetermined range 2906 of valuebased on a filter. The filter can be adjustable within a range of values2916, for example, via virtual 2904 and/or physical 2914 controls.

In some embodiments, a plurality of filters can be utilized forcustomizing a display image including palettized acoustic image data.FIG. 30 shows a display image showing a first indicator and a secondindicator. As described elsewhere herein, one or more filters can beapplied to the display image (e.g., via physical filter controls and/orvirtual filter controls) to customize the displayed data. In theillustrated example of FIG. 30 , the filtering includes establishing afirst filter range 3006 and a second filter range 3008. In some example,filter ranges can be adjustable within a range of values 3016, forexample, via virtual 3004 and/or physical 3014 controls.

Such filter ranges can represent any of a variety of parameters, such asfrequency, amplitude, proximity, etc. As shown, the first filter rangeand the second filter range are each associated with a color (which, insome examples, can be adjustable by a user), and indicators 3002, 3012are positioned at locations in the image at which corresponding acousticsignals meet the one or more filter conditions associated with eachfilter range. As shown, the first indicator 3002 represents acousticsignals that satisfies the first filter range 3006 (shown in a darkershade), while the second indicator 3012 represents acoustic signals thatsatisfy the second filter range 3008 (shown in a lighter shade). Thus, auser may be able to quickly identify locations in a scene havingacoustic data satisfying a variety of conditions at once, while alsoidentifying which locations satisfy which conditions.

In some examples, a display device, such as an acoustic imaging tool oran external display device, can include a virtual keyboard as an inputdevice, such as shown in FIG. 31 . FIG. 31 shows a display interfaceincluding an indicator 3102 representing one or more acoustic parametersof an acoustic signal in the scene. A virtual keyboard 3110 is includedin the display, which can be used to add alphanumeric information 3112to the display image. Utilizing such a virtual keyboard can allow a userto enter customize annotations, such as various inspection notes,labels, date/time stamps, or other data that can be stored with theimage. In various examples, the virtual keyboard can be used to add textthat is included in the image data and/or is appended to the image data,such as by being stored in metadata associated with a display image.

Various devices can be used to present a display image that includesvarious combinations of acoustic image data and other data, such asalphanumeric data, image data from one or more electromagneticspectrums, symbols, or the like. In some examples, a handheld acousticimaging tool can include a built-in display for presenting a displayimage. In other examples, information to be displayed, or data that isprocessed for generating a display (e.g., raw sensor data) can becommunicated to an external device for display. Such external devicescan include, for example, a smartphone, tablet, computer, wearabledevice, or the like. In some embodiments, the display image is presentedin combination with real-time electromagnetic image data (e.g., visiblelight image data) in an augmented realty-type display.

FIG. 32 shows a display embedded into eyewear 3210 that can be worn by auser. In some examples, eyewear can include one or more embedded imagingtools, such as described in U.S. Patent Publication No. 20160076937,entitled “DISPLAY OF IMAGES FROM AN IMAGING TOOL EMBEDDED OR ATTACHED TOA TEST AND MEASUREMENT TOOL,” and assigned to the assignee of theinstant application, relevant portions of which are incorporated hereinby reference. In some such examples, an integrated display can show areal-time display image 3220. For example, the display can showelectromagnetic image data (e.g., visible light image data)representative of the scene toward which the user is facing, and cansimultaneously display (e.g., via blending, overlay, etc.) one or moreadditional data streams, such as acoustic image data (e.g., includingindicator 3202) or the like, to provide added information to the user.In some embodiments, eyewear such as shown in FIG. 32 includes atransparent display screen, so that when no display image is provided tothe display, a user can view a scene directly with his or her eyesthrough eyewear rather than being presented with real-time visible lightimage data. In some such examples, additional data, such as acousticimage data, alphanumeric data, etc., can be displayed on theotherwise-transparent display in the user's field of view so that theuser views such data in addition to his or her view of the scene throughthe display.

As described elsewhere herein, in various examples, various datapresented in a display image can be combined in a variety of ways,including blending with other data streams (e.g., blending acousticimage data with visible light image data). In some examples, thestrength of the blending can vary between different locations within asingle display image. In some embodiments, a user can adjust theblending ratios of each of a plurality of locations (e.g., each of aplurality of indicators of detected acoustic signals) manually.Additionally or alternatively, blending can be a function of one or moreparameters, such as frequency, amplitude, proximity, etc.

In some embodiments, an acoustic imaging tool can be configured toidentify a degree to which the sensor array is pointing at each of aplurality of locations emitting detected acoustic signals and blendcorresponding acoustic image data with, for example, visible light imagedata, accordingly. FIG. 33A shows an exemplary display including a firstindicator 3302 and a second indicator 3304 representing acoustic signalsin an acoustic scene. In FIG. 33A, the acoustic sensor is pointing moredirectly at pipe 1, corresponding to the location of the first indicator3302, when compared to pipe 2, corresponding to the location of thesecond indicator 3304. As such, in the display scheme of FIG. 33A, thefirst indicator 3302 is displayed more prominently (e.g., has a higherblending coefficient or a lower transparency) than the second indicator3304. Conversely, in FIG. 33B the acoustic sensor is pointing moredirectly at pipe 2, corresponding to the location of the secondindicator 3304, when compared to pipe 1, corresponding to the locationof the first indicator 3302. As such, in the display scheme of FIG. 33B,the second indicator 3304 is displayed more prominently (e.g., has ahigher blending coefficient or a lower transparency) than the firstindicator 3302. In general, in some embodiments, an acoustic imagingsystem can determine a metric indicative of the degree to which thesensor is pointed at a given location (e.g., corresponding to anindicator in the acoustic image data) and adjust the blending ratioscorresponding to such locations accordingly (e.g., the greater thedegree of pointing corresponds to a higher blending ratio).

In some embodiments, acoustic imaging systems can include one or moreaudio devices to provide audio feedback based on the acoustic signalsreceived from the acoustic scene, for example, in addition to or as analternative to visualizing acoustic image data. A processor of anacoustic imaging system can be configured to generate an audio outputrepresentative of received acoustic data, and communicate the audiooutput to an audio device, which outputs audio feedback signals to theuser based on the generated audio output.

The one or more audio devices can include a speaker, headphones, aheadset or other wired or wireless audio device. In some examples, theaudio device can provide an audio or a heterodyne representation of theacoustic image data displayed on the display. For instance, in someexamples, the audio device can be configured to reproduce as audiofeedback signals the acoustic signals emanating from one or morelocations in an acoustic scene, and can provide such audio feedbacksignals individually or combined, for example, to reproduce acombination of sounds.

In some embodiments, the audio feedback can be provided to a usersimultaneously with a visual representation (e.g., a display imageincluding acoustic image data) of an acoustic scene to provide a usermultiple modes of analyzing the acoustic scene.

In some embodiments, a user or technician can aim the device based onthe audio feedback from the audio device, for example, to emphasize orcenter a sound of interest.

In some examples, in the event of a plurality of acoustic signals from acorresponding plurality of locations in an acoustic scene, an acousticimaging system including an audio device can cycle through imaged soundsand provide audio feedback on each imaged sound independently. In someembodiments, the acoustic imaging system can provide audio feedback to auser via the audio device corresponding to imaged sounds matching acriteria (e.g., based upon one or more selected parameters such asfrequency, decibel level, periodicity, distance, or the like).Additionally or alternatively, the system can be configured to provide acollective audio feedback signal representative of all detected acousticsignals.

In some examples, the audio device can provide a monophonic soundrepresentation of the scene or object of interest. Additionally oralternatively, the audio device can provide a stereophonic soundrepresentation of the scene or object of interest. In some embodiments,the audio device can toggle between monophonic and stereophonic soundrepresentation.

FIGS. 34A, 34B, 34C, and 34D show a variety of types of audio devices incommunication with an acoustic imaging device in an acoustic imagingsystem. Various examples include wired headphones (e.g., in-ear orover-ear headphones), wireless headphones/earphones (e.g., Bluetoothcommunication with acoustic imaging device), a wired and/or wirelessspeaker in communication with the acoustic imaging device, or anacoustic imaging device with built-in speakers on the sensor side and/orthe display/interface side. Combinations of such devices are possible,and in some such examples, a user may select one or more from aplurality of possible audio devices for outputting audio feedback.

FIG. 34A shows an acoustic imaging device 3402 having a port 3404 forinterfacing with wired headphones, such as 3406. FIG. 34B shows anacoustic imaging device 3412 including a wireless interface 3414 forwirelessly communicating with one or more wireless audio devices, suchas wireless speakers or headphones 3416. In some examples, an acousticimaging device includes integral speaks. FIG. 34C shows an acousticimaging device 3432 including integral speakers 3434 integrated into asensor side of the device. FIG. 34D shows an acoustic imaging device3442 including integral speakers 3444 integrated into a display side ofthe device. In general, acoustic analysis systems can include one ormore audio devices configured to output audio feedback signals to auser. Any combination of audio devices can be used, such as integralspeakers and wireless headphones, etc.

FIG. 35 shows an exemplary embodiment wherein an audio device providesaudio feedback to represent the displayed acoustic image data. Duringexemplary operation, an acoustic imaging device 3502 can initiallyreceive acoustic data and electromagnetic data from the scene andgenerate a display including acoustic image data and electromagneticimage data as described elsewhere herein. The user or technician candirect the acoustic imaging device such that a sound of interest islocated near the center point of the display screen. The center pointcan be represented in a variety of ways including crosshairs, a box, acircle, other shapes, or the like. When a visualized sound is at or nearthe center point of the display the acoustic imaging device can provideaudio feedback through the audio device for the user or technician, suchas via headphones 3506 wired to the acoustic imaging device 3502 viaheadphone jack 3504.

In the illustrated examples, acoustic imaging device 3502 detects asound 3510 in the scene, and outputs audio feedback 3520 based on thedetected sound 3510. Audio feedback 3520 provided via the audio device3506 can include reproducing the centered sound, including a reproducedone or more acoustic parameters, such as frequency, periodicity, etc. Insome examples, such parameters can be normalized to a more suitablescale for presenting to a user, such as providing audio feedback ofultrasound information in an audible frequency range and/or adjusting anintensity to a comfortable and/or audible level.

FIG. 36 shows an exemplary embodiment wherein an audio device providesselective audio feedback corresponding to a selected source of acousticsignals. In some examples, a user or technician can select which of aplurality of visualized sounds is to be used to generate audio feedbackthrough the audio device. In various examples, the user or techniciancan select one or more visualized sounds via a touch screen on adisplay, via one or more physical interfaces (e.g., buttons, etc.), viaan external device, or the like. In some examples, a user can draw orotherwise position one or more shapes or markers on the screen toidentify one or more locations from which to generate audio feedback.

In the illustrated examples, acoustic imaging device 3602 shows anacoustic image including indicators 3622, 3624, 3626 corresponding tosounds 3612, 3614, 3616, respectively. As shown in FIG. 36 , a userselects sound 3614 by selecting indicator 3624, for example, via atouchscreen. Audio device 3606, shown as wired headphones, outputs anaudio feedback signal 3634 representative of selected sound 3614. Theacoustic imaging device 3602 can be configured such that, if a userselects a different sound, for example, via selection of a differentindicator on the display, a corresponding audio feedback signal isoutput via the audio device 3606.

FIG. 37 shows another exemplary embodiment wherein an audio deviceprovides audio feedback to represent the displayed acoustic image data.Similar to FIG. 36 , the acoustic scene includes sounds 3712, 3714, and3716, and the corresponding acoustic image data is shown havingcorresponding indicators 3722, 3724, and 3726, respectively. In someexamples, a user or technician can receive audio feedback for all soundsin the acoustic image data or receive audio feedback only on soundswhich meet a certain criteria as described herein.

In some embodiments, a user or technician receives audio feedbacksignals via audio device 3706 representing each of sounds 3712, 3714,and 3716 simultaneously. Alternatively, the acoustic imaging device 3702can be configured to cycle through sounds and locations while providingaudio feedback for such locations one at a time. For instance, in anexemplary implementation, a user or technician first receives audiofeedback of 3622, then 3624, and then 3626. The acoustic imaging devicemay then return to sound 3612 to repeat the cycle, provides audiofeedback of all sounds, or the like. Thus, the acoustic imaging devicecan be used to sequentially output audio feedback signals 3732, 3734,and 3736 in response to detected sounds 3712, 3714, and 3716,respectively.

In various examples, a user can choose from one or more settingscorresponding to how audio feedback signals are provided in response toa scene including multiple sounds, including selecting from one or moreof: combining multiple sounds into a single audio feedback signal,cycling through a plurality of audio feedback signals each correspondingto one of the plurality of sounds individually, providing an audiofeedback signal corresponding to a single selected sound. As described,in some examples, the plurality of sounds combined and/or cycled throughare a subset of a plurality of sounds in the acoustic scene that meetone or more predetermined requirements. Additionally or alternatively,in some examples, an acoustic imaging system can cycle through aplurality of audio feedback signals in an order based on one or moreparameters, such as in order of descending intensity. In someembodiments, the acoustic imaging system can indicate on the displayimage which of a plurality of acoustic signals in an acoustic scene isbeing represented by a currently-provided audio feedback signal.

FIG. 38 shows an exemplary embodiment in which an audio device providesstereo audio feedback to represent the displayed acoustic image data.For example, FIG. 38 shows an acoustic imaging device 3802 and an audiodevice 3806 connected thereto. Sounds 3812 and 3826 are present in theacoustic scene, and are shown via indicators 3822 and 3826,respectively.

In some examples, the acoustic imaging device 3802 is configured toprovide stereophonic audio feedback to a user or technician, forinstance, in which acoustic signals from the left-hand portion of anacoustic scene are presented as audio feedback to a left-hand audiodevice 3850 and sounds on the right-hand portion of the scene arepresented as audio feedback to a right-hand audio device 3860. In theillustrated example, sound 3816 (and corresponding indicator 3826) isshown in the left-hand side of the acoustic scene, while sound 3812 (andcorresponding indicator 3822) is shown near the center of the scene. Insome systems employing stereoscopic audio feedback, audio feedbackcorresponding to sound 3812 is provided in both speakers (since it isnear the middle of the display screen), and audio feedback correspondingto sound 3816 is provided more to the left speaker 3850 than the rightspeaker 3860 (since it is closer to the left side of the displayscreen). In some examples, audio feedback corresponding to sound 3816can be heard only in the left ear.

In some embodiments, an acoustic scene can be divided into areas thatdictate how acoustic signals therein are represented in audio feedbacksignals. For instance, in an example, acoustic signals from a particulararea, such as the left-most quarter of the scene, are included in anaudio feedback signal output only from a left speaker, or have a greatermagnitude in an audio feedback signal output from a left speaker than inan audio feedback signal output from a right speaker. Similarly,acoustic signals from a particular area, such as the right-most quarterof the scene, are included in an audio feedback signal output only froma right speaker, or have a greater magnitude in an audio feedback signaloutput from a right speaker than in an audio feedback signal output froma left speaker. Acoustic signals more central to the scene can berepresented in audio feedback signals from both the right and leftspeakers.

In some embodiments, an acoustic imaging system is configured to receiveacoustic signals from a scene and generate acoustic image datarepresentative of such acoustic signals. In some embodiments theacoustic image data can be processed to determine how to balance audiofeedback signals between, for example, left and right channels increating stereoscopic audio feedback. For instance, in some examples,pixel locations of acoustic image data can help determine the relativepositioning of multiple sounds in a scene, and can be used to balancestereo audio feedback signals.

In the illustrated example, a left speaker 3850 outputs a first audiofeedback signal 3852 and a right speaker 3860 outputs a second audiofeedback signal 3862. In an example, the first audio feedback signal3852 represents only sound 3816 while the second audio feedback signalrepresents both of sounds 3816 and 3812.

While the illustrated example includes two speakers 3850 and 3860, insome examples, a system can include any number of audio output devicesthat can output audio feedback signals representative of receivedacoustic data. The audio feedback signals can be constructed based onthe locations of any number of acoustic signals detected within thescene.

In some examples, if the pointing of the acoustic imaging device were tochange so that, for example, sound 3812 were moved toward the right sideof the scene, the audio output would dynamically change so that only theaudio feedback signal 3862 would represent the sound 3812 on theright-hand side of the screen. Similarly, as sound 3816 moves toward thecenter of the screen, audio feedback signals 3852 and 3862 couldrepresent sound 3852.

Additionally or alternatively, as described elsewhere herein, dynamicoutputs based on a pointing of an acoustic imaging device can includeadjusting various aspects of a display image. For instance, as describedwith respect to FIGS. 33A and 33B, a transparency of or blending ratioassociated with acoustic image data can be adjusted based on therelative pointing of an acoustic imaging tool, such as a pointing of anacoustic sensor array. In some such embodiments, with respect to FIG. 38, in the pointing shown in the figure, indicator 3826 may be moretransparent than indicator 3822, since sound 3812 is more central withinthe scene. However, if the pointing of the acoustic imaging systemchanges so that sound 3816 moves toward the center and sound 3812 movestoward the right-hand side of the scene, indicator 3822 may become moretransparent than indicator 3826.

Thus, in some embodiments, an acoustic imaging system can produceoutputs, visual and/or audible, having dynamically changing intensitiesbased on the relative pointing of the system, such as the pointingdirection of an acoustic sensor array, an electromagnetic imaging tool,or both.

Various processes as described herein can be embodied as anon-transitory computer-readable medium comprising executableinstructions for causing one or more processors for carrying out suchprocesses. Systems can include one or more processors configured toperform such processes, for example, based on instructions stored inmemory integral to or external from the processor. In some instances,various components can be distributed throughout the system. Forinstance, a system can include a plurality of distribute processors,each configured execute at least a portion of the overall processexecuted by a system. Additionally, it will be appreciated that variousfeatures and functions as described herein can be combined into a singleacoustic imaging system, for example, embodied as a handheld acousticimaging tool or a distributed system having various separate and/orseparable components.

Various functionalities of components described herein can be combined.In some embodiments, features described in this application can becombined with features described in application No. PCT/US2019/043217entitled “SYSTEMS AND METHODS FOR PROJECTING AND DISPLAYING ACOUSTICDATA,” filed on Jul. 24, 2019, which is assigned to the assignee of theinstant application and which is incorporated herein by reference. Insome embodiments, features described in this application can be combinedwith features described in application No. PCT/US2019/043237 entitled“SYSTEMS AND METHODS FOR TAGGING AND LINKING ACOUSTIC IMAGES,” filed onJul. 24, 2019, which is assigned to the assignee of the instantapplication and which is incorporated herein by reference. In someembodiments, features described in this application can be combined withfeatures described in application No. PCT/US2019/043233 entitled“SYSTEMS AND METHODS FOR DETACHABLE AND ATTACHABLE ACOUSTIC IMAGINGSENSORS,” filed on Jul. 24, 2019, which is assigned to the assignee ofthe instant application and which is incorporated herein by reference.In some embodiments, features described in this application can becombined with features described in application No. PCT/US2019/043224entitled “SYSTEMS AND METHODS FOR ANALYZING AND DISPLAYING ACOUSTICDATA,” filed on Jul. 24, 2019, which is assigned to the assignee of theinstant application and which is incorporated herein by reference.

Various embodiments have been described. Such examples are non-limiting,and do not define or limit the scope of the invention in any way.

The invention claimed is:
 1. An acoustic analysis system, comprising: anacoustic sensor array comprising a plurality of acoustic sensor elementsthat receive acoustic signals from a scene and output acoustic databased on the acoustic signals; a processor in communication with theacoustic sensor array, the processor being configured to: receive theacoustic data from the acoustic sensor array; generate acoustic imagedata of the scene based on the acoustic data, the acoustic image dataincluding a visual representation of one or more acoustic signals in thescene; detect a plurality of acoustic signals in the scene; generate adisplay image comprising the acoustic image data, wherein the displayimage includes an indicator positioned at a location of each of theplurality of acoustic signals to identify a location of each acousticsignal in the scene; receive, via a user interface, an indication of aselected acoustic signal in the scene; generate an audio outputrepresentative of only the selected acoustic signal; and output theaudio output.
 2. The acoustic analysis system of claim 1, furthercomprising an audio device, wherein the processor is configured tooutput the audio output to the audio device.
 3. The acoustic analysissystem of claim 2, further comprising a housing supporting the acousticsensor array and the audio device, wherein the audio device comprisesone or more speakers supported by the housing, a receptacle forreceiving a headphone connector, or a wireless communication link forcommunicating with a wireless speaker or wireless headphones.
 4. Theacoustic analysis system of claim 2, wherein the audio device comprisesa first audio device and a second audio device, and wherein the audiooutput comprises a stereo audio output having a first audio output tothe first audio device and a second audio output to the second audiodevice such that the first audio device outputs a first audio feedbacksignal and the second audio device outputs a second audio feedbacksignal.
 5. The acoustic analysis system of claim 4, wherein theprocessor is further configured to: determine a location of the selectedacoustic signal in the scene; and wherein the first audio output and thesecond audio output are generated based on the location of the selectedacoustic signal in the scene.
 6. The acoustic analysis system of claim5, wherein: the display image includes a first area and a second area;when the selected acoustic signal is present within the first area, arepresentation of the selected acoustic signal in the first audio outputhas a greater magnitude than a representation of the selected acousticsignal in the second audio output; and when the acoustic signal ispresent within the second area, a representation of the selectedacoustic signal in the second audio output has a greater magnitude thana representation of the selected acoustic signal in the first audiooutput.
 7. The acoustic analysis system of claim 1, wherein the displayimage includes a visual representation of each of the plurality ofacoustic signals at respective locations in the display image; and theprocessor is configured to receive the indication of the selectedacoustic signal via the user interface by detecting a touchscreen inputat a location in the display image corresponding to the selectedacoustic signal.
 8. The acoustic analysis system of claim 1, wherein theprocessor is further configured to generate a plurality of audiooutputs, each of the plurality of audio outputs corresponding to one ofthe plurality of acoustic signals.
 9. The acoustic analysis system ofclaim 8, wherein the processor is configured to output each of theplurality of audio outputs simultaneously as a combined audio output.10. The acoustic analysis system of claim 8, wherein the processor isconfigured to output each of the plurality of audio outputs in asequence.
 11. The acoustic analysis system of claim 10, wherein theprocessor is configured to update the display image to indicate thelocation of the acoustic signal corresponding to the audio output beingoutput.
 12. The acoustic analysis system of claim 8, wherein theprocessor is configured to output audio outputs for only those acousticsignals satisfying a predetermined condition.
 13. The acoustic analysissystem of claim 1, wherein the processor is configured to determine oneor more acoustic parameters associated with an acoustic signal in thescene, and wherein the audio output is generated with one or moreacoustic parameters in common with the acoustic signal.
 14. The acousticanalysis system of claim 1, wherein the processor is further configuredto generate audio output that is representative of acoustic signalslocated in a center portion of the display image.
 15. The acousticanalysis system of claim 1, further comprising an electromagneticimaging tool configured to receive electromagnetic radiation from thescene and output electromagnetic image data representative of theelectromagnetic radiation, wherein the display image comprises acombination of the acoustic image data and electromagnetic image datareceived from the electromagnetic imaging tool.
 16. An acoustic analysissystem, comprising: an acoustic sensor array comprising a plurality ofacoustic sensor elements that receive acoustic signals from a scene andoutput acoustic data based on the acoustic signals; an audio deviceconfigured to output an audio feedback signal; and a processor incommunication with the acoustic sensor array and the audio device, theprocessor being configured to: receive the acoustic data from theacoustic sensor array; generate acoustic image data of the scene basedon the acoustic data, the acoustic image data including a visualrepresentation of one or more acoustic signals in the scene; generate adisplay image comprising the acoustic image data; and provide an outputrepresenting an acoustic signal within the scene, the outputrepresenting the acoustic signal having a dynamically changing intensityas a pointing of the acoustic sensor array is adjusted relative to alocation of the acoustic signal within the scene, wherein the displayimage includes a visual indication of the acoustic signal, and thevisual indication has a transparency that varies according to thedynamically changing intensity of the output; and wherein thetransparency of the visual indication of the acoustic signal variesaccording to the location of the acoustic signal within the scene, suchthat as the location of the acoustic signal moves away from a center ofthe scene, the transparency of the visual indication increases.
 17. Theacoustic analysis system of claim 16, wherein: the acoustic signal is afirst acoustic signal and the output is further representative of asecond acoustic signal within the scene; the display image includes afirst visual indication of the first acoustic signal and a second visualindication of the second acoustic signal, each of the first and secondvisual indications having a variable transparency in the display image;and the transparency of the first visual indication and the transparencyof the second visual indication respectively increase as the firstvisual indication and the second visual indication respectively moveaway from the center of the scene in the display image.
 18. The acousticanalysis system of claim 16, wherein the display image comprises a blendof the acoustic image data and electromagnetic image data representativeof electromagnetic radiation from the scene, and wherein varying thetransparency of the visual indication within the display image comprisesadjusting a blending ratio of the acoustic image data and theelectromagnetic image data.
 19. An acoustic analysis system, comprising:an acoustic sensor array comprising a plurality of acoustic sensorelements that receive acoustic signals from a scene and output acousticdata based on the acoustic signals; an audio device configured to outputan audio feedback signal; and a processor in communication with theacoustic sensor array and the audio device, the processor beingconfigured to: receive the acoustic data from the acoustic sensor array;generate acoustic image data of the scene based on the acoustic data,the acoustic image data including a visual representation of one or moreacoustic signals in the scene; generate a display image comprising theacoustic image data; and provide an output representing an acousticsignal in the scene, the output representing the acoustic signal havinga dynamically changing intensity as a pointing of the acoustic sensorarray is adjusted relative to a location of the acoustic signal withinthe scene, wherein: the audio device comprises a first audio device anda second audio device; the output comprises a stereo audio output havinga first audio output that is provided to the first audio device and asecond audio output that is provided to the second audio device; and anintensity of the first audio output and an intensity of the second audiooutput changes as the pointing of the acoustic sensor array is adjusted.20. The acoustic analysis system of claim 19, further comprising: anelectromagnetic imaging tool configured to receive electromagneticradiation from the scene and output electromagnetic image datarepresentative of the electromagnetic radiation; and a housingsupporting the acoustic sensor array and the electromagnetic imagingtool such that an orientation between the acoustic sensor array and theelectromagnetic imaging tool is fixed.